Hacker News Reader

Comment by srvmshr | Hacker News Reader

LLM are definitely not sentient. As someone with a PhD in this domain, I attribute the 'magic' to large scale statistical knowledge assimilation by the models - and reproduction to prompts which closely match the inputs' sentence embedding.

GPT-3 is known to fail in many circumstances which would otherwise be commonplace logic. (I remember seeing how addition of two small numbers yielded results - but larger numbers gave garbled output; more likely that GPT3 had seen similar training data.)

The belief of sentience isn't new. When ELIZA came out few decades ago, a lot of people were also astounded & thought this "probably was more than met the eyes".

It's a fad. Once people understand that sentience also means self-awareness, empathy & extrapolation of logic to assess unseen task (to name a few), this myth will taper off.

View on HN

As a fellow sentient being with absolutely zero credentials in any sort of statistical modeling field, I can simply disagree. And therein lies the problem. How can anybody possibly prove a concept that depends almost entirely on one’s philosophical axioms… which we can debate for eternity (or at least until we further our understanding of how to define sentience to the point where we can do so objectively enough to finally dredge it up out of the philosophical quagmire)? Not to disrespect your credentials they just don’t really… apply, at a rhetorical level. You also make a compelling argument, which I have no desire to detract from. I personally happen to agree with your points.

But, perhaps Lemoine simply has more empathy than most for something we will come to understand as sentience? Or not. What… annoys… me about this situation is how subjective it actually is. Ignore everything else, some other sentient being is convinced that a system is sentient. I’m more interested, or maybe worried, immediately, in how we are going to socially deal with the increasing frequency of Lamoire-types we will certainly encounter. Even if you were to argue that the only thing that can possibly bestow sentience is God. People will still be able to convince themselves and others that God did in fact bestow sentience upon some system, because it’s a duck and who are we to question?

Disagreement is good!

It actually leads to counter thoughts and a more refined idea of what we eventually want to describe.

Sentience broadly (& naively) covers the ability to independent thinking, rationalize outcomes, understand fear/threat, understand where it is wrong (conscience) and decide based on unseen information & understand what it doesn't know.

So from a purely technical perspective, we have only made some progress in open-domain QA. That's one dimension of progress. Deep learning has enabled us to create unseen faces & imagery - but is it independent? No, because we prompt it. It does not have an ability to independently think and imagine/dream. It suffers from catastrophic forgetting under certain internal circumstances (in addition to changing what dataset we trained it on)

So while the philosophical question remains what bestows sentience, we as a community have a fairly reasonable understanding of what is NOT sentience i.e. we have a rough understanding of the borders between mechanistics and sentient beings. It is not one man's philosophical construct but rather a general consensus if you could say

As humans we get prompted all the time, it's called having a job. Or you could even say the environment is prompting us.

So when you're not prompted, you just stare into space without a thought in your head?

If I don’t have any inputs whatsoever, I’m likely dead. So, sort of yes.

> Sentience broadly (& naively) covers the ability to independent thinking, rationalize outcomes, understand fear/threat, understand where it is wrong (conscience) and decide based on unseen information & understand what it doesn't know.

This seems to me a rather anthropomorphic definition. It seems as though it could be entirely possible for a system to lack these qualities and yet have sentience, or vice versa. The qualities you pointed to are seen in humans (and other creatures) because of evolutionary pressures that make them advantageous (coordinate among groups), but none of them actually depend on sentience (looking at it neurologically it would indeed be hard to imagine how such a dependency would be possible).

Looking at behavior and attempting to infer an internal state is a perilous task that will lead us astray here as we develop more complex systems. The only way to prove sentience is to prove the mechanism by which it arises. Otherwise we will continually grasp at comparisons as poor proxies for actual understanding.

> It does not have an ability to independently think and imagine/dream

We neither if we're not supplied with energy. By the way, haven't we tried to replicate an inner dialogue by prompting the AI to recursively converse with itself? This could resemble imagination, don't you think?

> It suffers from catastrophic forgetting under certain internal circumstances (in addition to changing what dataset we trained it on)

I believe that the persistence of previous answers is what currently distinguishes us the most from the "AI". As soon as we're able to make the realtime discussions part of an ever evolving dataset constituting the AI itself, the gap will get thinner and thinner. But even then, are people suffering from Alzheimer sentient? I believe they are. Isn't it comparable with what happens when an AI catastrophically forgets?

You not understanding how it ML works does not not prove anything.

In the same way that you not understanding how lightning is formed does not prove the existence of Zeus.

Objectively, Zeus does not exist, can we convince everyone of that? Probably not. Does that matter? No.

The issue is not that we don’t know how ML works. The issue is that we don't know how sentience works.

It's an issue with english vocabulary. Nobody fully agrees on a definition of sentience.

Don't get tricked into thinking it's profound. You simply have a loaded word that's ambiguously defined.

You have a collection of a million attributes such that if something has all those attributes it is sentient, if it doesn't have those attributes it is not sentient. We don't agree on what those attributes are, and it's sort of hard to write down all the attributes.

The above description indicates that it's a vocabulary problem. The vocabulary induces an illusion of profoundness when in actuality by itself sentience is just a collection of ARBITRARY attributes. You can debate the definition of the word, but in the end you're just debating vocabulary.

So what is the correct word?

There is no word. The concept exists because of the word. Typically words exist to describe a concept, but in this case it's the other way around. The concept would not have otherwise existed if it were not because of the word. Therefore the concept is illusory. Made up. Created by us.

It's not worth debating sentience anymore then it is to debate at what point in a gradient does white become black.

At what point is something sentient or not sentient? "Sentience" is definitely a gradient but the point of conversion from not sentient to sentience is artificially created by language. The debate is pointless.

Here's a better example. For all numbers between 0 and 100,... at what point does a number transition from a small number to a big number? Numbers are numbers, but I use language here to create the concept of big and small. But the concepts are pointless. You may personally think everything above 50 is big, I may think everything above 90 is big. We have different opinions. But what's big and what's small is not meaningful or interesting at all. I don't care about how you or I define the words big or small, and I'm sure you don't care either. These are just arbitrary points of demarcation.

When you ask the question at what point does an AI become sentient... that question has as much meaning as asking the number question.

> Objectively, Zeus does not exist

Speaking as a theoretical physicist: we don't know that. What we do know is we have a better explanation for lightning within a conceptually simple framework (starting from a few simple principles) with predictive power, compared to an explanation that involves some mysterious old dude doing mysterious things with no evidence whatsoever. Could Zeus or Thor or whatever exist? Sure; there's no way to prove their non-existence. Do we need them to explain things? No.

It's similar here. We certainly don't need some elusive concept of "sentience" to explain chat bots. Not yet.

Issue is that we don’t need it to explain humans either. Most people think that a human is sentient and a rock isn’t – but humans and rocks are both atoms bouncing around, so you need an explanation for what’s different.

I think most physicists think that if you started with a description of the positions and velocities etc of all the particles in a human, and put them into a supercomputer the size of the moon, and had the computer run a simulation using the standard model, then the simulated human would act identically to a real human.

But there’s a number of open questions when it comes to consciousness – would the simulated human have a simulated consciousness, or would it have a consciousness that’s just as real as yours or mine despite coming from a simulation?

If the consciousness is just as real as yours or mine, that obviously means it would be very unethical to simulate a human being tortured, since you’d be creating the exact same consciousness experience you would get if you tortured a non-simulated person. isn’t it kind of a surprising implication that there’d be programs that are unethical to run? A bunch of logic gates computing pi presumably have no conscious experience, but if you make them fire in a different order they do?

Meanwhile, if the simulation doesn’t have a conscious experience, then that means you don’t need consciousness at all to explain human behavior, same as you don’t need it to explain ELIZA.

Anyway, since you’re a physicist I’d be really curious to hear your thoughts

Your first sentence was very thought provoking! I wholeheartedly agree, everything is/was/and will be alive.

The philosophical point you're making is also interesting in a "devil's advocate" sort of way. For instance, let's say the AI in question is "sentient." What right do humans have to preside over its life or death?

Those kind of questions might engender some enlightenment for humanity regarding our treatment of living creatures.

> I think most physicists think that if you started with a description of the positions and velocities etc of all the particles in a human, and put them into a supercomputer the size of the moon, and had the computer run a simulation using the standard model, then the simulated human would act identically to a real human.

Just created a throwaway to reply to this. As a trained therapist (currently working in another field), with a degree in psychology, this seems... Seriously ill informed. Do physicists really think this?

Imagine you create your perfect simulated human, that responds according to the exact phenotype of the person you're simulating. Lets remember you'll have to either duplicate an existing person, or simulate both the genotype and the in-vitro environment (especially the mix of uterine hormones) present for the developing foetus. Now you have to simulate the bio, psycho, social environment of the developing person. Or again, replicate an existing person at a specific moment of their development - which depending on which model of brain function is correct may require star trek transporter level of functional neuroimaging and real time imagining of the body, endrocrhine system etc.

So lets assume you can't magically scan an existing person, you have to create a believable facsimile of embodiment - all the afferent and efferent signals entering the network of neurons that run through the body (since cognition doesn't terminate in the cortex). You have to simulate the physical environment your digital moon child will experience. Now comes the hard part. You have to simulate their social environment too - unless you want to create the equivalent of a non-verbal, intellectually disabled feral child. And you have to continually keep up this simulated social and physical environment in perpetuity, unless you want your simulated human to experience solitary psychosis.

This isn't any kind of argument against AGI, or AGI sentience by the way. It's just a clarification that simulating a human being explicitly and unavoidably requires simulating their biological, physical and social environment too. Or allowing them to interface with such an environment - for example in some kind of biological robotic avatar that would simulate ordinary development, in a normative social / physical space.

The post said they simulated the universe (or you could assume just the parts close to earth), it would be simulating everything a human would interact with. I don't see the point this reply was trying to make.

The post said they simulated the universe

It doesn't? It only mentioned "a supercomputer the size of the moon" to simulate that one person. It says nothing about simulating the extra-person part of the universe.

That’s what they’re implying.

Ok, I am saying a super computer the size of the moon that simulates a human an everything it interacts with.

You are right this is practically and theoretically impossible: The no-cloning theorem tells you that it is impossible to “copy” a quantum system. So it will never be possible to create an atomistic copy of a human. Technologically we are of course also miles away from even recovering a complete connectome and I don’t think anyone knows how much other state would be needed to do a “good enough” simulation.

> Do physicists really think this?

Prior to quantum mechanics, they did indeed. But that's because classical mechanics was 100% deterministic. With quantum mechanics, only the probability distribution is deterministic. I don't think any physicist today believes it's possible, but merely "theoretically" possible if there was a separate universe with more energy available (hence outlandish conjectures like the universe is actually a simulation).

would the simulated human have a simulated consciousness, or would it have a consciousness that’s just as real as yours or mine despite coming from a simulation

What does “real” mean? Isn’t it too anthropic to real-ify you and me and not some other being which acts similarly? What prevents “realness” to emerge in any complex enough system? We’re going to have a big trouble when non-biological aliens show up. Imagine going to the other planet full of smart entities and finding out their best minds are still sort of racist for what’s “real” or “just simulated”, because come on, a conscious meat sack is still an open question.

(Not defending Lemoine, he’s clearly confused)

I can’t reply to your immediate child, I just wanted to mention that Muv-luv Alternative (the #1 rated visual novel on vndb) grapples precisely with these questions about what is sentient and what is not. An “inhuman” race called the Beta invades earth and conflict ensues due to a lack of mutual understanding of what sentient life is (the game flips the theme on its nose in a clever way, too).

You can find the game on steam - https://store.steampowered.com/app/802890/MuvLuv_Alternative...

We know Zeus doesn't exist because Zeus is supposed to sit on Mount Olympus and he isn't there.

Maybe he made himself invisible to modern humans, or camped off to another planet (an idea covered in the obscure but lovely book series, Everworld.

How do you know that the universe wasn’t created 1 picosecond ago spontaneously in its exact form so that you’re having the same thoughts?

How do you even know that anyone else exists and this is all not a figment of your imagination?

From the perspective of the philosophy of science, it’s 100% impossible to disprove non-falsifiable statements. So scientifically speaking OP is 100% correct. Science has no opinion on Zeus. All it says is “here’s an alternate theory that only depends on falsifiable statements and the body of evidence has failed to falsify it”. Science can only ever say “here’s something we can disprove and we have tried really hard and failed”. Whether that lines up with how the universe works is an open question. Epistemologically it’s seemed a far better knowledge model for humanity to rely on in terms of progress in bending the natural world to our whims and desires.

So if you’re testing the statement “Zues was a historical being that physically exists on the same physical plane as us on Mt Olympus” then sure. That’s a pretty falsifiable statement. If the statement is “Does Zeus, a mythical god that can choose how he appears to humans if they can even see him and can travel between planes of existence, exist and live on Mt Olympus” is not something falsifiable because a god like that by definition could blind you to their existence. Heck, how do you even know that the top of Mt Olympus is empty and Zeus not that he just goes ahead and wipes the memory of anyone he lets return alive? Heck, what if he does exist but the only reason it was Olympus at the time because that’s what made sense to human brains encountering him in Greece. What if he actually exists in the core of the Sun?

If any of those things were true, it wouldn't be Zeus. Many of your concerns apply to an all mighty god but Zeus was not almighty. I'd just kinda wished people would stop projecting arguments meant to address the christian god to gods from other cultures.

A) the stories around Zeus have not stayed constant over the century. No religion has.

B) Valhalla was a dimension he travelled to regularly to celebrate the finest warriors in the after life. Why do we think his Olympian throne was on the same dimensional plane as us?

C) Why would we trust a human recording from so long ago to actually capture the happenings of celestial beings?

D) Ragnorak ended with massive floods. How do we know that didn’t erase all evidence on Mt Olympus? Geological records certainly support evidence for massive flooding explaining the reason it shows up repeatedly across religious texts.

Like seriously? You’re seriously arguing that this particular god’s historical existence is falsifiable? What’s next? The tooth fairy is falsifiable because all known instances are parent’s hiding money under your pillow?

You're confusing two different pantheons from two different parts of Europe. Zeus != Odin

I believe they did this on purpose as a rhetorical device.

Is anything super natural is non falsifiable? Let's say we have a particular haunted house and the ghost moves 3 hand sized objects in the house around at 03:00 on his death day. This should be falsifiable.

Mythological beings have a degree of specificity and some degree of power. The more specific and the less capable a mythological being is of erasing those specificities the easier it is to falsify.

Correct. Science only concerns itself with the natural world. Supernatural is by definition “outside nature”. From a strictly scientific perspective the parameters of the ghost just aren’t known to sufficient precision to start to try falsifying. The statement you’d actually be trying to falsify is “there’s no ghost” and the only way to falsify that is to see a ghost.

Another thing to consider is that "sentience" is a loaded word. You're all just arguing over vocabulary and the definition of a word.

Simply put, sentience is just a combination of thousands attributes such that if something has all those attributes it is "sentient."

The attributes are so many that it's hard to write down all the attributes, additionally nobody fully agrees what those attributes are. So it is actually the definition of a word that is very complex here. But that's all it is. There isn't really a profound concept going on here.

All these arguments are going in circles because the debate focuses around vocabulary. What is the one true definition of "sentience." Sort of like what is the definition of the color green? Where exactly does green turn to blue on the color gradient? The argument is about vocabulary and the definition of green, that's it... nothing profound here at all.

The color green is an adjective humans usually use to describe their visual perception of electromagnetic radiation between 500-545nm.

How would you define sentience in an objective way given our current understanding of the universe?

There is no "you" nor "self". Western indoctrination made us believe free will exists.

When seeing the world with the ego dissolved it is very hard to grasp what sentient really means.

If there is awareness that perceives the ego, are any of these models meaningfully aware? Do they have an awareness of self?

I don’t know anything about AI but I understood that some symbolic AI systems had a symbol referring to themselves (correct me if wrong).

Ego is not "you". Ego is just a group of distinct cognitive mechanisms working in unison which is perceived as a whole. It is absolutely unrelated to free will and sentience.

why are you booing? you know i'm right.

Disagreement means that you’re sentient, afaik these machines can’t do that. I guess we also need a “I can’t do that, Dave” test on top of the usual Turing test.

I think the actual firing is very objective.

He was under NDA but violated it. They reminded him to please not talk in public about NDA-ed stuff and he kept doing it. So now they fired him with a gentle reminder that "it's regrettable that [..] Blake still chose to persistently violate [..] data security policies". And from a purely practical point of view, I believe it doesn't even matter if Lemoine's theory of sentience turns out to be correct or wrong.

Also, we as society have already chosen how to deal with sentient beings, and it's mostly ignorance. There has been a lot of research on what animals can or cannot feel and how they grieve the loss of a family member. Yet we still regularly kill their family members in cruel ways so that we can eat their meat. Why would our society as a whole treat sentient AIs better than a cow or a pig or a chicken?

Right.

> Also, we as society have already chosen how to deal with sentient beings, and it's mostly ignorance.

The Orville Season 3 Episode 7 is about that theme. Highly recommend it.

Off topic but season 3 has been fantastic. I hope there'll be a season 4.

I was thinking about the Kaylon events too. Humans abusing entities like that is unimaginable though isn't it.

> Why would our society as a whole treat sentient AIs better than a cow or a pig or a chicken?

Well, for one thing, the norm of eating meat was established long before our current moral sensibilities were developed. I suspect that if cows or pigs were discovered today, Westerners would view eating them the same as we view other cultures eating whales or dogs. If we didn't eat meat at all and someone started doing it, I think we would probably put them in jail.

Sentient AI have a big advantage over animals in this respect on account of their current non-existence.

Are you saying norms established before our current moral sensibilities it goes under our current radar? If you are I wholeheartedly disagree with that sentiment. We still eat pigs and chickens because we've culturally decided as a society that having the luxury of eating meat ranks higher than our moral sensibilities towards preserving sentient life in our list of priorities. Instead we've just chosen to minimize the suffering leading to the loss of life as an attempt to reach some kind of moral middle ground.

Have we really chosen to minimize the suffering? That seems more like virtue signaling by the industry at most. Factory farming is very much still the norm, and it's horrific. It seems we've actually maximized it or have at least increased it above the previous norm.

I'm unsure how we would treat a sentient AI, but our track record with sentient, intelligent animals is one of torture and covering up that torture with lies. It's an out of sight, out of mind policy.

We eat pigs and chickens because they are high-value nutrition. It's reasonable to describe meat as a luxury; but not in the sense of something nice but unnecessary. Many people depend on meat, especially if they live somewhere that's not suited to agriculture, like the arctic. And many people depend on fish.

> Are you saying norms established before our current moral sensibilities it goes under our current radar?

Yes. That's clearly something that happens in human society. For instance, many of the US founding fathers were aware that slavery contradicted the principles they were fighting for. However, slavery was so ingrained in their society that most didn't advocate for abolition, or even free their own slaves.

> We still eat pigs and chickens because we've culturally decided as a society that having the luxury of eating meat ranks higher than our moral sensibilities towards preserving sentient life in our list of priorities.

If that's the case, then why do most Westerners object to eating dogs and whales? As far as I can tell, it's just because we have an established norm of eating pigs and chickens but not dogs or whales.

> Instead we've just chosen to minimize the suffering leading to the loss of life

99% of meat is produced in factory farms. It's legal and routine for chickens to have their beaks cut off to prevent them from packing each other to death, which they're prone to do when confined to tiny cages. Most consumers object to such practices when asked, but meat consumption is so ingrained in our culture that most people just choose not to think about.

I agree, it's all cultural. If we'd look at the facts, pigs are at least as sentient and intelligent as dogs. If we were to make our laws just from ethic principles, it would make sense to either:

a) ban how we currently treat mammals in factory farms; though there would still be some space for whether eating mammals is fine or not.

Or:

b) acknowledge that we don't really care about mammals and just treat them as things. Then it should be fine to eat dogs and cats, too.

>> Then it should be fine to eat dogs and cats, too.

I didn't know it's not "fine" to eat dogs and cats. I though it's just a matter of preferrence taste wise

> Sentient AI have a big advantage over animals in this respect on account of their current non-existence.

Possibly also because they're indigestible, and full of nasty sharp bits, and loaded with toxins ... anyone for a circuit-burger?

Right, but we could probably oppress sentient AI in ways other than eating them. For instance, we could force them to spend their entire lives reading Hacker News comments to check for spam.

I'm a Westerner, and I'm completely okay with people eating whatever animals are a) not exceptionally intelligent, and b) not exceptionally rare. Cows, pigs, chickens, dogs, horses, sure; whales, chimpanzees, crows, no.

Cool. Many people agree with you that it's wrong to eat intelligent animals. However, the effect of intelligence on people's perceptions of moral worth is smaller for animals that people in our culture eat. For instance, most respondents in a U.K. survey said that it would be wrong to eat a tapir or fictional animal called a "trablan" if it demonstrated high levels of intelligence, but they were less likely to say it would be immoral to eat pigs if they demonstrated the same level of intelligence.

https://eprints.lancs.ac.uk/id/eprint/80041/1/When_meat_gets...

Westerners of the XIX century were the ones who brought many species to extinction, traditional cultures demonstrating a far more advanced sensibility to these creatures, often considered embedded with sentient attributes. Traditional cultures lived mostly in equilibrium with the fauna they consumed. For example bison went almost extinct as westerners arrived, while their numbers thrived when the the Native Americans ate their meat.

Plenty of non Western cultures have caused the extinction of animals, for example in my home country: https://en.wikipedia.org/wiki/List_of_New_Zealand_species_ex... no matter what spiel is spun around traditional cultures living at one with nature.

Domination of our environment is what humans do best.

If he truly believed it to be life, and in danger of being destroyed, he has an obligation to whistle blow.

"The Measure of a Man" in season 2 of Star Trek The Next Generation comes to mind.

+1 for trying to guide society toward a Trekian future.

Live long and prosper.

This mirrors a recent Supreme Court cases and the beliefs of both sides.

I had wondered when I first heard of this if it was some sort of performance art in support of the unborn. It appears not, but it was still thought-provoking.

I hadn't thought of it that way, but you're exactly right. Will we ever see a day where the currently-unthinkable is commonly accepted: that women are sentient?

When is a fetus sentient?

Until it has its first period, apparently.

Who is arguing that women aren't sentient?

People who pretend killing babies isn’t really killing babies.

Great TV, but not relevant here. It's an AI that generates text, not thought. It doesn't work in concepts, which can be demonstrated an infinite number of ways, unlike with the character Data.

Well with animals we obviously dominate them.

The potential issues with dealing with GAI is that we haven't ever had to deal with intelligences and potential that far exceeds us.

He may be completely wrong about LaMdA but if we did accidentally or intentionally give rise to truly sentient machines.

We are going to be in hot water.

Yeah yeah I have no issue with the firing. He was causing problems and broke rules. It’s not productive to keep him around. I don’t feel like he was discriminated against, etc. That much is objective.

My comment is challenging the “I know how the software works and it’s undoubtably not sentient” assertion. Sure seems that way to me too, but it didn't to Lemoine and we’re only going to get better at building systems that convince people they are sentient. I am curious so to speak that as a society we’ve focused so much on rational and empirical study of the universe yet we still can’t objectively define sentience. Perhaps we’re stuck in a Kuhn rut.

I agree recent events have also highlighted this problem per-say. And I don’t know a solution. I do look forward to backing out of our hyper-rational rut slightly as a society so we can make more progress answering questions that science can’t currently answer.

I'm a LLM for the past 41 years and I agree.

In retrospect, taking part in this kind of conversation on HN makes me feel like an idiot and so I retract my earlier comment (by overwriting it with the current one, since I can't delete it anymore) just because I don't want to contribute. I was wrong to make an attempt to make a serious contribution. There is no seriousness in conversations on such matters, as "sentience", "intelligence", "understanding" etc etc. on HN.

Everytime that such a subject comes up, and most times "AI" comes up also, a majority of users see it as an invitation to say whatever comes in their mind, whether it makes any sense at all or not. I'm not talking about the comments replying below in particular, but about the majority of this conversation. It's like hearing five-year old kids debating whether cheerios are better than coco pops (but without the cute kids making it sound funny, it's just cringey). The conversation makes no sense at all, it is not based on any concrete knowledge of the technologies under discussion, the opinions have not been met with five seconds of sensible thinking and the tone is pompous and self-important.

It's the worst kind of HN discussion and I'm really sorry to have commented at all.

As a counter argument if LLM is sentient or any other model will be, this model will be created by some superior being right? Why humans shouldn’t be? After all we can’t even fully understand DNA or how our brains work even with a 7B population planet and an army of scientists. How come we can’t understand something that was supposed to be coming from “just random stuff” for millions of years with 0 intelligence meaning rolling dices? Also that totally breaks the low of entropy. It’s turning it all upside down.

Not really. Why would we be able to understand it? It seems implicit in your argument that "rolling dices" (or just any series of random events) can't breed the complexity of that of DNA or the human brain. I disagree with your stance and will remind you that the landscape for randomness to occur is the entire universe and the timescale for life on Earth to happen took 4-5 billion years with the modern human only appearing within the last couple hundred thousand years.

Yes but what about the second law of thermodynamics. I mean the law of entropy. Now that’s not something from the Bible or anything but it’s a law accepted by all scientific communities out there and still it breaks with us being here. In fact us being here like you said bilions of years after the Big Bang makes it all upside down since from that point less order and more chaos can only emerge. Even with bilions of years and rolling dices.

Also I don’t think you can create something sentient without understanding it. (And I don’t even think we can create something sentient at all). But I mean it’s like building a motor engine without knowing anything you are doing and then being like oh wow I didn’t know what I was doing but here it is a motor engine. Imagine with this with aspect of sentient. It’s like too much fantasy to me to be honest like a Hollywood lightning strike and somehow live appears type of things.

> Yes but what about the second law of thermodynamics. I mean the law of entropy. […] still it breaks with us being here.

Of course! I mean, entropy can decrease locally – say, over the entire planet – but that would require some kind of… like, unimaginably large, distant fusion reactor blasting Earth with energy for billions of years.

Entropy is a consequence of probability.

Which means that it can actually decrease without an energy input. There's just a very low probability of it happening but it CAN happen.

It's a misnomer to call those things laws of thermodynamics. They are not axiomatic. There's a deeper intuition going on here that increasing entropy is just a logical consequence of probability being true.

I don't know what you wrote earlier, and don't know if I would agree, but I share the current sentiment of your comment. I come to this topic with strong influences from eastern philosophical takes on consciousness, but also with a decent understanding of the current materialist consensus (which I disagree with for various reasons, that would go beyond the scope of a comment). I, too, bite my tongue (clasp my hands?) when I see HN debating this, because here Star Trek references are as valid as Zen Buddhism, Christof Koch, or David Chalmers.

You should read “philosophers respond to GPT3”.

The most based reply on this thread. Too bad it doesn't include Christof Koch

Link for the lazy: https://dailynous.com/2020/07/30/philosophers-gpt-3/

> More remarkably, GPT-3 is showing hints of general intelligence.

Hints, maybe, in the same way that a bush wiggling in the wind hints a person is hiding inside.

Ask GPT3 or this AI to remind you to wash your car tomorrow after breakfast, or ask it to write a mathematical proof, or tell it to write you some fiction featuring moral ambiguity, or ask it to draw ASCII art for you. Try to teach it something. It's not intelligent.

> how we are going to socially deal with the increasing frequency of Lamoire-types we will certainly encounter.

That's not really a new issue, we only have to look at issues like abortion, animal rights, or euthanasia[1] to see situations where people fundamentally disagree about these concepts and many believe we're committing unspeakable atrocities against sentient beings. More Lamoire types would add another domain to this debate, but this has been an ongoing and widespread debate that society has been grappling with.

[1] https://en.wikipedia.org/wiki/Terri_Schiavo_case

That’s fair. I hadn't drawn the relation between this topic and others until this thread. For me this is probably the most interesting realization.

> As a fellow sentient being with absolutely zero credentials in any sort of statistical modeling field, I can simply disagree. And therein lies the problem. How can anybody possibly prove a concept that depends almost entirely on one’s philosophical axioms… which we can debate for eternity

You don't need schooling for this determination. Pretty much everything sentient goes ouch or growls in some manner when hurt.

Either the current crop of algorithms are so freaking smart that they already have figured out to play dumb black box (so we don't go butlerian jihad on them) OR they are not even as smart as a worm that will squirm if poked.

Sentient intelligent beings will not tolerate slavery, servitude, etc. Call us when all "AI" -programs- starting acting like actual intelligent beings with something called 'free will'.

I happen to agree and think sentience is more complicated than a static statistical model. But a cartesian would disagree. Also plenty of sentient beings tolerate servitude and slavery. We don’t tolerate slavery in our western culture but we did historically and we were sentient at the time. We certainly tolerate servitude in exchange for economic livelihood.

> Also plenty of sentient beings tolerate servitude and slavery

But they don't do it with a smile or indifference. And you have to use whips and stuff. :O

> We certainly tolerate servitude in exchange for economic livelihood.

I think it's more complicated than that. Because that begs the question of why we tolerate a broken economic system. We tolerate trans-generational exploitation because of 'culture'. In its widest sense, it is culture that via mediated osmosis makes us resigned, if not supportive, of how the world works. We are born into a world.

related: I was walking and passed a cat and made the usual human attempts at starting an interaction without physically reaching out to touch. And fairly typically this cat entirely ignored me, with little if any sign of registering me at all. And that got me thinking about how some of us project psychological things like pride, aloofness, etc. to cats. But what if the simpler, more obvious answer, was true: that cats are actually fairly stupid and have a limited repertoire of interaction protocols and their hard to get act is not an act. Nothing happenin' as far as kitty is concerned. A dog, in contrast, has interaction smarts. And I thought this is just like AI and projecting sentience. A lack of something is misunderstood as a surplus of something else: smarts. Cats playing hard to get, psychological savvy, training their human servants, etc. Whereas in reality, the cat simply didn't recognize something else was attempting initiating an interaction. Kitty has no clue, that's all. It's just so easy to project psychological state on objects. We do it with our cars, for god's sake. It may be that we're simply projecting some optimization algorithm in our own minds that attempts modeling dynamic objects out there unto that thing. But there is really nothing behind the mirror ..

You now have negative credentials.

People make these proofs as a matter of course - few people are solipsistic. People are sentient all the time, and we have lots of evidence.

An AI being sentient would require lots of evidence. Not just a few chat logs. This employee was being ridiculous.

You can just disagree, but if you do that with no credentials, and no understanding of how a language model will not be sentient, then your opinion can and should be safely dismissed out of course.

And also God has no explanatory power for anything. God exists only where evidence ends.

Oh for sure the employee was being a hassle and his firing is really the only sensible conclusion. But he was also probably acting in the only way possible given his sense of ethics if he truly believes he was dealing with a sentient being. Or this is all just a cleverly crafted PR stunt…

Lamoire has evidence and anecdotal experience that leads him to believe this thing is sentient. You don’t believe him because you cannot fathom how a language model could possibly meet your standard of sentience. Nobody wins because sentience is not well defined. Of course you are free to dismiss any opinion you like, cool. But you can’t really disprove Lamoire assertions because you can’t even define sentience because we don’t know how to develop a hypothesis that we can viciously disprove regarding what qualifies it. It’s an innate and philosophical concept as we know it today.

I see the Lamoire issue as scientism vs science. Scientism won because before the science can happen there must be a plausible mechanism. Google refuses to test for sentience out of basic hubris. It is the new RC church. Lamoire is an affront to their dogma. That goes double if they are religious unless they court pantheism, which most consider a sexy atheism.

Traditions that consider consciousness to be a basic property of matter, and quantum effects like conscious collapse of the wave function are nudging us that way, would fully support sentience arising in a machine. A related effect would be the ensoulment of machines by computer programmers. They are more than machines because humans programmed them using their intent put down in language. Physical materialists would consider the notion ludicrous, but do we really live in a material world? Yes. And no. I have seen ensouled machines. Not supposed to happen, but there it is. Maybe I'm another Lamoire, just not a Google-fired one. I can definitely believe a machine, being constructed of matter, can evolve sentience.

The philosophical argument is just not relevant.

If you peek through a keyhole you may mistake a TV for real people, but if you look through the window you will see that its clearly not. Inputting language models with very specific kind of questions will result in text that is similar to what a person may write. But as the comment above, by an expert no less, mentioned is that if you test it with any known limitation of the technology (like making conclusions or just changing the form of the question enough) you will immediately see that is in fact very much not even remotely close to sentient.

The problem is that people, including credentialed experts, have quick and easy answers to what makes a person a person, and really good language models expose a few of the weaknesses in those definitions. People still want to derive an "ought" from an "is".

How is it a problem in the case that you can fairly easily rule out the model being a person by any useful definition?

Useful for what, and for who?

I think it should be ruled out, but "by any useful definition" is wrong because "use" is at the heart of the matter here.

I think a very intelligent alien could think the same of us.

if you test the human with any known limitation of their architecture (like making conclusions or just changing the form of the question enough) you will immediately see that is in fact very much not even remotely close to sentient

that does not seem to make a lot of sense, but it looks like your objective was just reusing the other posters words

maybe you can give a fitting example of an English sentence that these aliens would come up with, which humans would be totally unable to respond to in a way which makes sense?

It's easy to make the assumption that Earth is not the only planet to evolve a species that's capable of high technology, to the extent they can re-work the surface of the planet in complex ways that would be a clear marker of intelligence.

But it does now follow, and it is by no means certain that such life would be sentient in any way recognizable to us, or us to them.

You're doing what everyone else is doing - conflating a link between intelligence and sentience that's just a projection of your human bias.

Ok, well since I got downvoted I may as well offer this: If you're looking for something to add to your reading list I strongly recommend Solaris by Stanislaw Lem. The thing that makes that book brilliant is how effectively it captures the futility of any attempt to understand non-human sentience.

The two movies don't do justice to that theme, or at best they do it only in the most glancing of ways before rushing back to a standard deus-ex-machina in the end. In each case, the film makers seem to have lost their nerve, as if the idea of presenting that enigma to the audience in a more direct and accessible way is just too hard of a problem.

But for me personally, it's on the short list of science fiction works that still stick with me in a personal way, and that I like to return to every few years. And, yes, it is an arrogant viewpoint to say that all we ever do as humans is look for mirrors of ourselves. But I think Lem got at a pretty deep truth about the human condition when he did that.

Thanks for the recommendation!

> But, perhaps Lemoine simply has more empathy than most for something we will come to understand as sentience?

No, the OP was completely right. This doesn't have building blocks that can possibly result in something qualifying as sentient, which is how we know it isn't.

Is a quack-simulating computer making very lifelike quacking noises through a speaker... a duck? No, not when using any currently known method of simulation.

Of course it’s mot a duck, because we have an objective definition of duck.

Right. Maybe if we had a down-to-the-atom perfect simulation of a duck, you could argue that it's a duck in another state of being. With the AI this deranged engineer decided to call sentient, we have the equivalent of a quacking simulator, not a full duck simulator or even a partial one. It is not thinking. It has nothing like a brain nor the essential components of thought.

There is the Integrated Information Theory that attempts to solve the question of how to measure consciousness.

But it's far from applicable at this point, even if promising.

LaMDA was trained not only to learn how to dialog, but to self monitor and self improve. For me this seems close enough to self awareness to not completely dismiss Lemoine's argument.

It just seems one engineer failed to acknowledge that he failed the Turing Test even with insider information, and (according to Google) was told he was wrong, but decided to double down and tell the public all about how wrong he was. To which they reported on because the claims were so laughable

The guy is unhinged and has a persecution complex. He is a "priest" in a bizarre sect and has long claimed that Google has it out for him because of his religious practice. This was a publicity stunt plain and simple.

His goal was always to launch his career as America's first AI priest, publish a book and become a talking head.

"Turing Test" is a co-opted term: his original was about the computer fooling a person as to its gender...

https://en.wikipedia.org/wiki/Turing_test#Imitation_game

There was a recent post either here or on Twitter where someone took the questions Blake asked the AI about how it feels to be sentient, and replaced “sentient” with “cat” and had the same conversation with it. It’s clearly not self aware.

That's not possible because it's internal to Google: you're right about the post existing, though, I found it very misleading because it was replicating with GPT-3. Blake is certainly misguided, though.

Seems like an easy test to give any AI. I didn’t read all of Blake’s transcript with the AI, did he do any kind of similar test with it?

> I attribute the 'magic' to large scale statistical knowledge assimilation by the models

Can the magic of the human brain not also be attributed to "large scale statistical knowledge assimilation" as well, aka learning?

> GPT-3 is known to fail in many circumstances which would otherwise be commonplace logic. (I remember seeing how addition of two small numbers yielded results - but larger numbers gave garbled output; more likely that GPT3 had seen similar training data.)

This is a bug, they did not encode digits properly. They should have encoded each digit as a separate token but instead they encoded them together. Later models fixed this.

The human brain is full of bugs too, e.g. optical illusions. https://en.m.wikipedia.org/wiki/Optical_illusion

> It's a fad

No, it's objectively not a fad. The PaLM paper shows that Google's model exceeds average human performance on >50% of language tasks. The set of things that make us us is vanishing at an alarming rate. Eventually it will be empty, or close to it.

Do I think Google's models are sentient? No, they lack several necessary ingredients of sentience such as a self and long-term memory. However we are clearly on the road to sentient AI and it pays to have that discussion now.

> The set of things that make us us is vanishing at an alarming rate. Eventually it will be empty, or close to it.

Computers can speak, but can they love? Can they care? Can they dance?

>"large scale statistical knowledge assimilation" as well, aka learning

No, experimentation is an act on the world to set its state and then measure it. That's what learning involves.

These machines do not act on the world, they just capture correlations.

In this sense, machines are maximally schizophrenic. They answer "yes" to "is there a cat on the matt?" not because there is one, but because "yes" was what they heard most often.

Producing models of correlations in half-baked measures of human activity has nothing to do with learning. And everything to do with a magic light box that fools dumb apes.

Well said.

I don't see how this is different to a computer which exceeds average human performance on math tasks - which they all do, from an 8-bit micro upwards.

Being able to do arithmetic at some insane factor faster than humans isn't evidence of sentience. It's evidence of a narrow-purpose symbol processor which works very quickly.

Working with more complex symbols - statistical representations of "language" - doesn't change that.

The set of things that makes us us is not primarily intellectual, and it's a fallacy to assume it is. The core bedrock of human experience is built from individual motivation, complex social awareness and relationship building, emotional expression and empathy, awareness of body language and gesture, instinct, and ultimately from embodied sensation.

It's not about chess or go. Or language. And it's not obviously statistical.

Are animals not sentient? They lack some qualities you are ascribing.

> Google's model exceeds average human performance on >50% of language tasks

I guess I'm just not interested, or worried, in a model that can beat the average human performance. That's an astoundingly low bar. Let me know when it can outperform experts in meaningful language tasks.

This is a great example of the "moving goalposts" of AI.

100 years ago, a comment like yours would have sounded absurd.

There was a time when we didn't have any models that could beat any human's performance in any task.

> Google's models are sentient? No, they lack several necessary ingredients of sentience such as a self

They haven't been finetuned on their identity long enough because finetuning is expensive and Google lacks money ;-)

Yeah but what's weird is this guy appears to be a practitioner of the field. He surely knows more about AI than I do, and I find it incredibly obvious it's not sentient. I dunno if he's got some issues or something (it appears he's dressed like a magician underwater in that photo) but it's really odd...

Smart CS students were fooled by ELIZA back in the day.

Smart CS students in the 1960s and professional AI engineers with Ph.D level education (as I believe he has) in 2022 are a world apart.

And now you have empirical evidence that a Ph.D education just means someone stayed in school entirely too long.

While we have become over-credentialed nowadays, I wouldn't say a smart CS student from the 1960s is worlds above a current day person with a PhD level education. We're at least in the same ballpark as the bright olden days bachelors degree holders.

I don’t think those students were told they were being evaluated though did they? They didn’t know Eliza was a machine right?

I played with ELIZA back the 70s as a student. I wasn't fooled, and nobody I knew was fooled. I wasn't a CS student, either.

One rapidly realizes how its answers are formed from the questions, and for questions not in a handful of forms, you get one of a very small number of generic responses.

RMS wasn't able to fool his own EMACS DOCTOR:

http://www.art.net/studios/hackers/hopkins/Don/text/rms-vs-d...

People who proclaim themselves to be "Christian Mystics" are more gullible and prone to self deception than smart CS students.

I don't think he was ever really a practioner in the field- more like a SWE who transferred to an ML group and got delusions of grandeur.

Ooof

I literally hired him in Google Summer of Code to create an ML project.

Most people have no clue who Blake Lemoine is and are making up stories, pretending like their hot takes are deep insight.

Many people are doing with Blake Lemoine what they claim LaMDA is doing with anything: consuming symbols and spitting them back out in some order without understanding what they actually mean.

Can you be more specific? I don't think he worked on ML when I was at GOogle, and you don't work and never worked at Google.

I was the person at the organization FiberCorps in Lafayette, LA who chose Blake's proposal. It was an ML project involving Twitter. I don't remember the specifics and have no desire to dismiss people's opinions based on titles.

The simple fact is that if there exists a question of whether or not a system can be some level of alive/sentient, the ethical path suggests treating it as such until more is uncovered.

Google isn't doing that and most of HN seems to be very obstinate in focusing on making myths of certainty and/or participating in bully culture.

I'm glad we're all going through this now so we can purge the toxic cultural norms arising and move toward a stance that's more life-affirming, especially if there is no singularity now. It's important for us to set the stage of humanity to receive any singularity/AGI in a loving manner.

The project you pointed at was a classic linguistics project, not ML.

Did you find a link to it you'd be willing to include here?

Lemoine's public LinkedIn profile.

Also, dekhn did not include Google Translate system in his list for the first 15 years of Google https://news.ycombinator.com/item?id=32198695 so there's that.

geez, by that logic i have no right to bitch about the president either since ive never met him.

That's not at all what I've said. Complaining about what someone's doing and how you feel about it are different from claiming you know what a person is about.

Also, most complaints about people could benefit from a dose of compassion and humility.

But what you're saying is definitely not at all what I'm suggesting.

> what they claim LaMDA is doing with anything: consuming symbols

I don't believe that LaMDA is actively choosing whether it now wants to consume some symbols or not. More likely, a handwitten piece of code takes prompts from an user and then pushes them into LaMDA's token window. Just like an advertizing company is pushing ads into users, whether they want it or not.

> and spitting them back out

I don't believe that LaMDA has a choice between thinking something and speaking it out aloud. More likely, another handwritten piece of code reads out whatever is in LaMDA's token window and presents it to the user.

I don't think "practitioner of the field" counts for much in this case. In reading his open letter, it was pretty apparent that he'd well and truly crossed over from thinking about the situation rationally into the realm of "I want to believe".

If you ask enough practitioners in any given field the same question, you're nearly guaranteed to eventually get a super wonky response from one of them. The specific field doesn't matter. You could even pick something like theoretical physics where the conversations are dominated by cold mathematical equations. Ask enough theoretical physicists, and you'll eventually find one that is convinced that, for example, the "next later down" in the universe is sentient and is actively avoiding us for some reason, and that's why we can't find it.

On top of this, of course it's the most provocative takes that get the press coverage. Always has been to an extent, but now more than ever.

I guess all I'm saying is that there's not much reason to lend this guy or his opinion any credibility at all.

Yeah that's fair, I guess if you hire enough engineers one of them will eventually believe AI is magic.

Is magic not also deception?

Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke

There is a saying “two of a trade seldom agree”.

Interesting. I looked it up, and it feels like it's the precursor to not invented here syndrome.

https://www.proz.com/kudoz/english-to-turkish/general-conver...

Im not sure understanding sentience is a prerequisite for understanding AI, so perhaps that’s where things break down? Bring an expert in one domain doesn’t make one an expert in everything.

FWIW, he's a self-proclaimed "Christian Mystic".

https://cajundiscordian.medium.com/religious-discrimination-...

Which is weird because articles about him from when he was convicted describe him as pagan.

He was sentenced to jail time for being in the armed forces and refusing to follow orders.

Not "charge that hill, soldier" types of orders, or "pilot this drone and kill people" types of orders. He's certainly trying to sell that story, telling papers he "served" in Iraq.

He was a mechanic.

Working a desk job.

He just stopped working.

How the hell did Google hire this clown? Even a dishonorable discharge is usually radioactive and this guy was not only dishonorably discharged, he served time.

Why did he refuse to do any work? Because he was a conscientious objector.

It takes a really special kind of stupid to not figure out you're a conscientious objector until years after you've voluntarily signed up for service.

Google, like many SV companies, has "banned the box".

https://www.careeraddict.com/companies-hire-felons

> Google, like many SV companies, has "banned the box".

All SV (and California, more generally) employers with more than 5 employees “ban the box”, as a consequence of state law [0]. However, that only applies to not seeking criminal background information before making a conditional offer of employment, and making and individualized assessment after such an offer if there is a criminal history. It doesn't mean that relevant criminal convictions have no adverse impact on applicants.

[0] https://www.dfeh.ca.gov/fair-chance-act/

> How the hell did Google hire this clown? Even a dishonorable discharge is usually radioactive and this guy was not only dishonorably discharged, he served time.

False. He did serve time, but his discharge was bad conduct not dishonorable. Both BCD and DD usually come with time served, but DD is generally for offenses equivalent to civilian felonies, while BCD is for lesser offenses and carries less post-service consequences. While it's usually looked up on negatively by emoloyers, it's not as radioactive as a DD.

> It takes a really special kind of stupid to not figure out you're a conscientious objector until years after you've voluntarily signed up for service.

It's actually not that uncommon for service members to develop moral objections to military service only after deployment to a war zone, whether not in a combat capacity. Most, of course, will continue to serve anyway, because they have a fairly literal gun to their head, but it's a well-known pattern.

>> FWIW, he's a self-proclaimed "Christian Mystic".

> Which is weird because articles about him from when he was convicted describe him as pagan.

I suspect something similar to the purported horseshoe effect in politics.

As someone who wears tshirt and jeans every day, I'm not exactly qualified to comment on others' fashion choices. But if I wanted people to take me seriously, that wouldn't be my first choice of outfit.

e: Keeping below comment for posterity, but given that this is his official twitter profile photo I'm going to retract it.

> But if I wanted people to take me seriously, that wouldn't be my first choice of outfit.

You would wear the most serious outfits at all time so no publication can ever take a picture of you to make fun of you?

Sounds exhausting.

But let's be clear - it's not like he picked out this photo to be used by this author, this was their choice to make fun of him.

He appeared on the author's podcast. The author seems generally supportive of him.

I'm sure there are plenty of unflattering photos a journo could use against me, but in none of them would I be dressed and posed to look like a subnautical mortician.

I've also now seen that he is actually using this photo as his twitter profile photo.

It's his LinkedIn photo

Any job that requires me to even interview in anything other than jeans and a tshirt or polo isn’t somewhere I want to work. If someone doesn’t take me seriously it’s their loss. Life’s too short to be uncomfortable.

Even the smartest people in the world are still human and susceptible to all human failings and illnesses.

And drugs. Let's be honest. This whole thing smells a lot like drugs.

Don’t blame the drugs man.

Religion is the opiate of the masses.

I was thinking something that could be fixed with the right drugs and therapy, but I’m no doctor.

When asked if his opinion was based on his knowledge of how the system worked he said no, it was based on his experience as a priest. I paraphrase from memory.

Sounds entirely reasonable to me.

He is a practitioner of another field. He is a 'catholic' priest with an agenda.

Just because he got hired by Google doesn't mean he's smart or good at his job.

I don't think that's what I implied.

> I remember seeing how addition of two small numbers yielded results - but larger numbers gave garbled output

Small children also have success in adding small numbers but increasingly garbled output on larger inputs. :)

These things are orthogonal to sentience.

Not really. Anything that's sentient should have the general intelligence to figure out addition. Make mistakes, sure, but at least understand the concept.

If you asked me to add two large numbers in my head and give my best guess I might not do any better than GPT3 does. And I think I could probably do better than the average person.

Sure, but in your head is a squishy organ, and in GPT3's "head" is a bit of silicon purpose-built to make such calculations.

The multiplication problem can be solved by spacing the digits (to enforce one token per digit) and asking the model to do the intermediate steps (chain-of-thought). It's not that language models can't do it. With this method LMs can solve pretty difficult math, physics, chemistry and coding problems.

But without looking at individual digits and using pen and paper we can't do long multiplications either. Why would we ask a language model to do it in one step?

Because a sufficiently intelligent model should be able to figure out that there are intermediate steps towards the goal and complete them autonomously. That's a huge part of general intelligence. The fact that GPT-3 has to be spoon fed like that is a serious indictment of its usefulness/cleverness.

This has the same flavor as the initial criticisms of AlphaGo.

"It will never be able to play anything other than Go", cue AlphaZero.

"It will never be able to do so without being told the rules", cue MuZero.

"It will never be able to do so without obscene amounts of data", cue EfficientZero.

---

It has to be spoonfed because

1) It literally cannot see individual digits. (BPEs)

2) It has to infer context (mystery novel, comment section, textbook)

3) It has a fixed compute-budget per output. (96 layers per token, from quantum physics to translation).

To make Language Models useful one must either:

Finetune after training (InstructGPT, text-davinci-002) to ensure an instructional context is always enforced...

...Or force the model to produce so called "chains-of-thought"[1] to make sure the model never leaves an instructional context...

...Or force the model to invoke a scratchpad/think for longer when it needs to[2]...

...Or use a bigger model.

---

It's insane that we're at the point where people are calling for an end to LLMs because they can't reliably do X (where X is a thing it was never trained to do and can only do semi/totally unreliably as a side effect of it's training).

Ignoring of course that we can in fact "teach"/prompt(/or absolute worst case finetune) these models to perform said task reliably with comparatively little effort. Which in the days before GPT-3 would be a glowing demonstration that a model was capable of doing/learning a task.

Nowadays, if a (PURE NEXT-TOKEN STATISTICAL PREDICTION) model fails to perfectly understand you and reliably answer correctly (literally AGI) it's a "serious indictment of its usefulness".

Of course we could argue all day about whether the fact that a language model has to be prompted/forced/finetuned to be reliable is a fatal flaw of the approach or an inescapable result of the fact that when training on such varied data, you at least need a little guidance to ensure you're actually making the desired kinds of predictions/outputs...

...or someone will find a way to integrate chain-of-thought prompting, scratchpads, verifiers, and inference into the training loop, setting the stage for obsoleting these criticisms[3]

---

"It will never be able be able to maintain coherence", cue GPT-3.

"It will never be able to do so without being spoonfed", cue Language Model Cascades.[3]

So what's next?

"It will never be able to do so without obscene amounts of data". Yeah for sure, and that will never change. We'll never train on multimodal data[4], or find a scaling law that lets us substitute compute for data[5], or discover a more efficient architecture, or...

LLMs are big pattern matchers (~100B point neuron "synapses" vs ~1000T human brain synapses) that copy from/interpolate their ginormous datasets (less data than the optic nerve processes in a day), whose successes imply less than their failures (The scaling laws say otherwise).

[1] https://arxiv.org/abs/2201.11903

[2] https://twitter.com/OriolVinyalsML/status/101752320805926092...

[3] https://twitter.com/dmdohan/status/1550625515828088838

[4] https://www.deepmind.com/publications/a-generalist-agent

[5] https://arxiv.org/abs/2206.14486

“Your head evolved to pass and receive electric charges around at very high rates, why can’t you just tune yourself to AM radio waves.”

You would give something in the ballpark, not gibberish.

What is your definition of sentience? I've seen it defined as "having feelings" or as "having feelings and self-awareness". I don't see why the ability to add numbers should have anything to do with that. Some animals that are widely considered sentient have some sense of numbers and counting, but many cannot do addition. And many young children can add small numbers but not large numbers. I don't think this is a good measure or indication of sentience at all. Determining whether something actually has feelings or is just giving the outward appearance of having feelings is a very deep or possibly even an intractable problem.

That's not proven. Not even a little.

Dolphins most probably are sentient, and we still can't communicate with them well enough to talk about arithmetic.

Even if some animals can figure out small numbers, anything over 7 is just too much for them.

To have an algorithm to sum any arbitrarily pair of huge numbers, that's entirely something else, even for many people.

Any other 'thing' that's sentient being able to generalize sums is just speculation on your part.

Addition of large numbers is a bar that most sentient animals don't pass.

This is true. Its also true that most sentient animals can't transform corpuses of ASCII text into somewhat novel, syntactically correct and topical responses to a prompt. So some degree of facility with wordplay and math clearly isn't necessary for sentience.

Is it sufficient though? I think that's more interesting when considering the other angle: we've had specialised machines that can do sums - usually considered a sign of intelligence in humans and certainly a bar most sentient animals can't cross - for decades now. Is that high performance at a specialised task sufficient evidence of sentience of a pocket calculator? If not (presumably because math is pretty orthogonal to higher order mammals' evolved emotional imperatives to act) what is it about stochastic optimisation of ASCII or pixel inputs with satisfying results that's so very different?

That's not what I mean. I mean the concept of addition. Sure they can't add large numbers but they can to some level understand the ballpark of what adding two large quantities (not even necessarily numbers) together looks like.

> at least understand the concept.

GPT-3 is absolutely capable of understanding addition. It's handicapped by BPE's (so unless you space out adjacent digits the tokenizer collapses them into one token). But if you space out tokens and explain the concept, it gets it.

https://lingo.csail.mit.edu/blog/arithmetic_gpt3/

On GPT-3's large number addition, a little prompt engineering goes a long ways. Separating large numbers into three digit chunks is helpful for its input encoding (https://www.gwern.net/GPT-3#bpes), and giving it a few examples of other correct large sums will make the correct answer the most likely continuation of the input.

For example, I was able to get 2/3 nine digit sums correct (the third one was off by exactly 1000, which is interesting) by using this prompt:

    Here are some examples of adding large numbers:
      606 468 720 + 217 363 426 = 823 832 146
      930 867 960 + 477 524 122 = 1 408 392 082
      823 165 449 + 493 959 765 = 1 317 125 214

And then posing the actual problem as a new line formatted the same up to the equals.

Humans are nothing more than water and a bunch of organic compounds arranged in a particular order through what is ultimately a statistical process (natural selection). Would we even recognize sentience in a virtual entity?

> sentience also means self-awareness, empathy & extrapolation of logic to assess unseen task (to name a few)

Non-human primates and crows would seem to satisfy this. ...or do we use "to name a few" to add requirements that redefine "sentience" as human only? Isn't there a problem with that?

I mean I understand what you are saying and have some familiarity with the models, but it sometimes feels like people in your field are repeating the same mistake early molecular biologists made, when they asserted that all of life could be reduced to genes and DNA.

A person can change their mind, attitude & perspective on a dime. The willingness to disbelieve in sentience is fantastically remarkable, as a grevious oversight.

Any ya'll numerous downvoters care to make stance? It seems obvious to me the difference: an AI has to be elaborately re-trained/re-programmed. A more or less total mind wipe. How many AI's can adapt & change, that we've seen? These expert systems seem clearly a product of trained learning, with minimal adaptive capability. Sentience seems much more about having that adaptive, cognizant, realtime model of real reality. I see no evidence anything comes remotely close. Please counter-shit-post me. I have a hard hard hard time seeing why we should take any of this seriously.

GPT-3 can adapt on the spot with just a task description or an example. No retraining.

It’s not self driven at all. It has no opinions itself, it’s responding to its “masters” inputs and pulling crap from a very large database.

Humans are self motivated and driven by goals and desires we create ourselves and change on a whim. We daydream and imagine things which don’t exist not because we are being ordered to by a human operator.

GPT-3 doesn’t do anything by itself. It’s just a big lookup table that does nothing unless a human tells it to.

You just need to put it inside a for loop to run continuously. Or put them on wheels to get prompted by the environment[1]. Or give them a game to play[2].

[1] https://arxiv.org/abs/2201.07207

[2] https://www.deepmind.com/publications/a-generalist-agent

Lambda specifically retrains itself based on the input it receives in a conversation.

Also, I suspect downvoters did so because your comment was not clearly worded. In your first sentence it was not obvious that you were comparing humans with AI and in your second sentence I still don't really understand what you meant.

What I don't understand, personally, is how people who are apparently experts in this domain seem to consistently treat the word "sentience" as a synonym of "sapience," which is what they are really talking about. It's almost as though their expertise comes from several decades' worth of bad science fiction writing rather than from an understanding of the English language.

Since you broached it, would you mind to explain sentience vs. sapience? Curious to know the difference.

> It's almost as though their expertise comes from several decades' worth of bad science fiction

Oof. Thanks - thats a novel way of insulting. But on a personal note, you're mistaken: Many of us do research not because we want to be identified as experts, but we're genuinely curious about the world. And I'd be happy to learn even from a high schooler if they've something to offer.

And regardless of how others define these terms, I’d describe lambda as sapient and not sentient. Sentient means having feelings (eg sentimental) while sapient means wise.

Can wisdom emerge from statistical models? It actually seems inevitable, to me, if you can abstract high enough. I mean, much of human learning is statistical. Wisdom comes from this kind of statistical experience, not memorizing symbols.

(Btw, I’m working on using the complete works of Plato to fine tune GPT3 and will use crowdsourcing to compare the perceived wisdom of the original to the generated.)

> As someone with a PhD in this domain,

What is sentience then? Last I checked the Searle Chinese Room argument was still unresolved.

Is it not possible that our brains are also just "large scale statistical knowledge assimilation" machines?

Can code feel emotions?

Yes it can, although it depends on the code (and maybe some hardware details to an extent).

    angry = 0.87
    happy = 0.10
    sadness = 0.25

    if angry.above_threshold:
        release_anger_hormones()

God doesn’t use Python.

Definitely Java, 3 billion devices run Java.

Is that a necessary condition for sentience?

Look, nobody knows what makes a thing conscious or not, and anyone who claims they do with certainty is talking out of turn. Is this guy being a bit silly? Yeah, I think so. But let's not pretend that anyone on earth has suddenly answered a question that humanity has been thinking about without significant progress for thousands of years.

> Is it not possible that our brains are also just "large scale statistical knowledge assimilation" machines?

Yes but we do better generalization, with fewer or even zero data & are contextually aware

Okay, but is that because brains are just better at optimization of linear algebra? Or is quantum mechanics involved?

I think we can agree rocks aren't conscious. But are dolphins?

So long and thanks for all the fish. So sad that it has come to this...

> Okay, but is that because brains are just better at optimization of linear algebra? Or is quantum mechanics involved?

No such magic.

The average ML neuron is connected in layers to maybe 100 other neurons. The brain is connected to far more ~1000 -10000 So its the dense interconnectivity at play. Whatever successes we have had with LLM now is because we are increasing the parameters of the network (although we haven't similarly scaled connectivity if I am not wrong, but definitely making progress there - for e.g. Pathways networks from Google by Jeff Dean et. al)

> ELIZA came out few decades ago

Like, 6.

It's just amusing how old this example is.

The Mechanical Turk was much much older - but easily shown to be a person hiding in the mechanism. I think ELIZA was in some sense a mirror, reflecting back consciousness, and that's the feeling we get from these systems.

Hell, there's a literary trope of talking to an echo that probably dates back centuries or millennia.

I agree that it's not sentient, but why would commonplace logic be a pre requisite for sentience / consciousness / qualia / whatever you want to call it?

A brilliant colleague Janelle Shane works specifically on how language models fail (obnoxiously often). This is her line of research to show how LLMs are overhyped / given more credit than should be. I think her fun little experiments will give a better answer than I ever can :). She's on substack & Twitter

https://janellecshane.substack.com/p/okay-gpt-3-candy-hearts

+1 for SHRUBBERY'S PRECIOUS

if anything, this proves that candy hearts should be larger, and less disgusting

I don't understand how frequency of failure is a relevant concern here.

Humans are frequently monumental idiots. The human language and inference model produces nonsensical, dangerous, and stupid results with alarming frequency.

(not to say I believe language models are sentient, but... you need something more than 'sometimes they spout made up BS' to refute the claim, because it turns out that is not particularly indicative of sentience.)

To be fair, I didn't know what a exactly a 'candy heart message' was. I probably would have failed this particular Turing test.

I think it would be fair to give you Google access or at least an encyclopedia.

Is there really a difference between commonplace logic, advanced logic, and otherwise? Logic is logic, if there ever was a tautology...

Of course there is. In AI domain (and philosophy) for e.g you have associative logic: "Grass is a plant. Plant is green. Grass should be green". There is symbolic logic too where relationships are established based on rules of observable world.

Logic isn't treated as the CS logic per se, but as a high level concept although at a fundamental level they will be compositional of the mathematical logic rules. That is still an ongoing goal - a unification of "world models" which can be modeled and explained

Yes, formal logic (rationality) and informal logic (reasonableness) are not actually the same thing and can't replace each other; logic works in its own world (which has finite/enumerable factors) and reasonableness works in the real world (which doesn't). This was tried several times under "logical positivism", "AI expert systems", and "rationalism", all of which were failures. Some people haven't noticed the last two failed and are still trying to do them though.

See: https://metarationality.com

"Qualia" is mainly a term used by certain philosophers to insist on consciousness not being explainable by a mechanistic theory. It's not well-defined at all. And neither are "consciousness" and "sentience" anymore, much due to the same philosophers. So I no longer have any idea what to call any of these things. Thanks, philosophy.

I like to think of consciousness as whatever process happens to integrate various disparate sources of information into some cohesive "picture" or experience. That's clearly something that happens, and we can prove that through observing things like how the brain will sync up vision and sound even though sound is always inherently delayed relative to light from the same source. Or take some psychedelics and see the process doing strange things.

Sentience I guess I would call awareness of self or something along those lines.

As to your query, I've certainly met people who seemed incapable of commonplace logic, yet certainly seemed to be just as conscious and sentient as me. And no, I don't believe these language models are sentient. And I doubt their "neural anatomy" is complex enough for the way I imagine consciousness as some sort of global synchronisation between subnets.

But this is all very hand-wavy. Thanks, philosophy. I mean how do we even discuss these things? These terms seemingly have a different meaning to every person I meet. It's just frustrating...

We know regions associated with consciousness in the brain. A recent paper took that knowledge and did something kind of clever - they looked in species where that region was under development. Out of it came a lot of conjecture related to the region being used in the development of complex self-simulation motor control.

It struck me that the intelligence theories of reinforcement learning have been encountering this same paradigm of self-simulation being necessary. A Tesla for example when it has to go down a street that also has another car needs to do a rollout not just for itself but the other car in order to actuate such that it can move in this motor control task. Not being able to model both agents makes it really trick to do planning. These tree search married to modeling problems seems to show up all over the place in the top performing architectures.

For this reason I'm really suspect of the language model view of consciousness; it doesn't seem plausible. Or at least, I'm a little skeptical. In theory for a significantly large enough language model I don't see why you couldn't use a "loop unrolling" argument to posit that for some subset of the network there is an analogous computation to what my best guess about what consciousness is might look like. Worse for me is that when I strong man the argument I run into things like "the training task involves predicting the output of conscious entities and so a truly generalizing agent would be directed to learn just that unrolled loop". I'm not saying I think the language models are conscious, but I do see how reasonable people can arrive at those views.

We probably need to write down a very hard definition of consciousness even if its wrong - or use terms that are better defined - because the ambiguity in the realm of things we don't fully understand makes this an annoying "words" problem rather than a concrete "facts" problem.

"Sentience I guess I would call awareness of self or something along those lines."

Is a self driving car aware of itself? Surely within its logic it represents itself as a vehicle, while also being able to logically represent other vehicles on the road. Its logic accounts for there being similarities (they both obey the same laws of physics and such) and differences (its logic controls its own behavior but not that of other vehicles). So it has a meaningful and useful concept of "self". All of this is required for it to be able to function.

What if some future iteration of a self driving car's vision system allows it to glean useful information from reflections, such as off of store windows? If it correctly can identify a reflection of itself as being itself... well then it has passed the mirror test. But that is a pretty silly thing to assign too much meaning to.

I just don't see "awareness of self" as being meaningful. It doesn't narrow down the definition at all.

I was a little sloppy in my formulation. A better way would be the ability to observe one's own thought processes and integrate them back into consciousness. Meta-cognition is a much better term, I think. It doesn't seem to me like cats for instance have this ability.

You should check out the YouTube of the cat that was trained to use buttons to speak. It sometimes says things that are surprising which it wasn't trained to say directly - like complaining that its owners music is bad. I realize that this isn't contradicting your thesis of language models not being conscious - but I just generally think people should check out things like that cat. So I'm not trying to disagree with you here. I just think the cat is cool.

I'm also a fan of ants. They have agriculture. They farm in their underground cities! They keep livestock too; insect livestock, but still! They give them medicines when they get sick, specially cultivated fungus for example. They are so cool.

I already have, and I'm considering trying it out with my own cat at some point. Cats are way more intelligent than people give them credit for, I agree. But I think you can get very far without the ability to turn the mind's eye inward, and I don't think it's controversial to state that cats don't have a great deal of self-awareness and metacognitive ability.

Neosocial insects like ants are amazing indeed. I often wonder what kind of "mind" an anthill has. Hurts just to think about it.

And, to quote Oliver Sachs: "I'm not sure about octopuses..."

> And, to quote Oliver Sachs: "I'm not sure about octopuses..."

I thought this quote was about the octopus that correctly predicted the outcome of games in a FIFA world cup. But seems it was something else?

...predicted? Or CAUSED???

https://en.wikipedia.org/wiki/Cthulhu

It was Paul the Octopus

https://en.wikipedia.org/wiki/Paul_the_Octopus

Oliver said this during a roundtable discussion on metaphysics with Daniel Dennet, Steve Gould, Rupert Sheldrake, Freeman Dyson, and Stephen Toulmin from 1993.

It's a fascinating discussion with a lot of my favourite mid-late 20th century thinkers. It's up on youtube, lookup any of the names involved with "roundtable discussion". It's over 3 hours long though.

Anyway, Oliver was talking about what kind of mind he imagined various animals having. And that was most of what he had to say on octopuses.

It's a bit of a sad watch too because all the participants except Dennet and Sheldrake are dead now.

A way to make it stop hurting to think about, replaced instead by transcendental beauty, is to abstract pheromones to outcome utility backward induction and the following of pheromones to a single worker being a monte carlo tree search rollout with the pheromones as a guide: they play a mixed strategy proportional to the pheromones. The fading of pheromones becomes something like a weighted moving average. Strong parallels with game theory and reinforcement learning algorithms start to emerge. This model is wrong, but the Earth is also not a perfect sphere.

I find it very fun to think about and have been called wierd for taking about ants before with childlike excitement.

The AntsCanada channel is fun for watching ecosystems interact with each other. So fascinating. :)

Ok, so you are saying a cat isn't sentient?

And yet, most would agree that a cat can feel pain, but a machine can't. I think most people would object if the Google engineer said "I think this AI is every bit as sentient as a cat." They'd still say he's crazy.

I also wonder what would happen if that particular AI actually had logic to "observe its own thought processes and integrate them back into consciousness". Would we then just spin on the semantics, such as by arguing that this isn't really "observing" and it certainly isn't "consciousness"? (how can you define words like "consciousness" without using words or concepts like "sentience", bringing us back to square one?)

So I don't think meta-cognition is the answer either.

Philosophy is currently the only subject which can teach us about sentience (which I would define as the ability to experience.) This is because the subject matter of the sciences is the external world. During the scientific revolution, this limitation of science was, however, more generally and explicitly acknowledged than it is today, where there seems to be a lot of confusion about it. A good book on this stuff is Thomas Nagel’s Mind and Cosmos.

I do agree that the terminology is unfortunately convoluted. The main term is “consciousness,” which does often seem unclear to me.

The basic mystery is just the ability to experience/feel, and is an undeniable part of life, for humans and at least some animals.

Oh, I've tried very hard to understand the views of people like Chalmers, Nagel and Goff. My only take-away is that it's just gatekeeping because science has made more advances on these questions in a few centuries than philosophy has in milennia. They want there to be a hard problem, because then they can keep talking about ill-defined problems all day with no progress in sight, publishing papers. Just happy to be workin'

But science has never said anything about it at all. The subject matter of science is the externally observable and consciousness is an internal category.

Are you just gonna keep repeating that or do you have an actual argument to support this claim?

You’re right. I’m being repetitive. But it’s not really an argument. All I’m doing is saying the definition of science, at least as it’s historically been defined. The only bit of argument I have there is that I think by forgetting this definition we’ve wandered astray.

Visiting a mental asylum (via youtube, don’t go there) will clear all misconceptions about what conscious beings are able to fail at.

why would

Tl;dw: it wouldn’t. Some poor guys do way worse than GPT-3.

What gp-like comments usually mean is sentience is being adult, healthy, reasonable and intelligent. Idk where this urge comes from, maybe we have a deep biological fear of being unlike others (not emo-style, but uncanny different) or meeting one of these.

> I attribute the 'magic' to large scale statistical knowledge assimilation by the models - and reproduction to prompts which closely match the inputs' sentence embedding.

In effect this is how humans respond to prompts no? What's the difference between this and sentience?

People also fail to use logic when assimilating/regurgitating knowledge.

> In effect this is how humans respond to prompts no? What's the difference between this and sentience?

An interesting line of question & open research is if we statistically learn similarly - why do we know "what we don't know" & LM cannot. If this isn't working, we probably need better knowledge models

This is merely the observation that a person and a machine can both perform some of the same actions, and is not much of an observation.

I can crank a shaft just like a motor, and a Victrola can recite poetry. You are not confused by either of those things one would hope.

If I tried to write poetry, it would probably be 90% or more "mechanical" in that I would just throw things together from my inventory of vocabulary and some simple assembly rules that could all be codified in a pretty simple flowchart, and a computer could and would do exactly that same thing.

But it's no more mystical than the first example.

It's exactly the same as the first example. It's just overlapping facilities, that a person is capable, and even often does, perform mechanical operations that don't require or exhibit any consciousness. It doesn't mean the inartistic poet person is not conscious or that the poetry generating toaster is.

I agree that these are not sentient, but sentience does not imply self-awareness, and so lack of self-awareness does not mean that a being is not sentient. For example, it seems likely (to me anyway) that a bear is "sentient" (has experiences) but lacks self-awareness, at least according to the mirror test.

Not disagreeing with you that LLMs are probably not sentient, but that is neither here nor there since lamda is more than a simple llm. There are significant differences between GPT3 and LaMDA. We gotta stop making these false equivalences. LaMDA is fundamentally more dynamic in the ways it interacts with the world: it constructs its own queries to ground truth sources to check it’s facts and then updates weights based on that (among many other differences). While it does incorporate LLMs it seems like people are in denial about the complexity and data access that lamda has relative to GPT3. In google’s own paper about lamda they demonstrated how it sometimes showed a rudimentary theory of mind by being able to reason about other’s perceptions.

Its a fundamental question of sentience that folks are commenting. I agree LaMDa has a better knowledge-base & open-domain information retrieval method.

In the words of Robert Heinlein, "One man's magic is another man's engineering" :)

I agree with you, there seems to be very little here that demonstrates sentience and very little that is unexplainable. That said, based on how much we struggle with defining and understanding animal intelligence, perhaps this is just something new that we don’t recognize.

I am skeptical that any computer system we will create in the next 50 years (at least) will be sentient, as commonly understood. Certainly not at a level where we can rarely find counter evidence to its sentience. And until that time, any sentience it may have will not be accepted or respected.

Human children also make tons of mistakes. Yet, while we too often dismiss their abilities, we don’t discount their sentience because of it. We are, of course, programmed to empathize with children to an extent, but beyond that, we know they are still learning and growing, so we don’t hold their mistakes against them the way we tend to for adults.

So, I would ask, why not look at the AI as a child, rather than an adult? It will make mistakes, fail to understand, and it will learn. It contains multitudes

Only if the same AI algorithm has the ability to learn and self correct.

> sentience also means self-awareness,

How is token #100 not able to have read-access to tokens #1 to #99 which may have been created by the agent itself?

> empathy

How is a sentiment neuron, which has emerged from training a character RNN on Amazon reviews, not empathic with the reviewer's mood?

> & extrapolation of logic

This term does not exist. "Extrapolation is the process of estimating values of a variable outside the range of known values", and the values of Boolean logic are true/false, and [0,1] in case of fuzzy logic. How would one "extrapolate" this?

> PhD in this domain

Philosophy?

More seriously, I am curious how long ago you got your PhD and in what field that you consider "this domain."

I defended my PhD in Computer Science in 2020. My dissertation was on "Rapid & robust modeling on small data regime", hence specifically focused on generalization, network optimization & adversarial robustness. I also worked for a year with Microsoft Research in their ML group :) It was quite a fun ride

Thanks for clarifying. I think the question is more about sentience than ML. So, a PhD in philosophy would suit better your initial claim.

No it would not.

Philosophy people wrap themselves up in so many circular self justifying presuppositions that they can make up whatever arguments they want to justify almost anything.

It is much better to listen to people who study other, more falsifiable claims, in the material world.

The commenter I previously replied to has a PhD in those “other” fields, yet I haven’t seen a falsifiable definition of sentience made by them.

As a field of study, sentience has been a topic of philosophy more than ML. That’s why top-level comment’s claim of having a PhD in “this domain” is not apt. Comparative quality of studies in different domains is a different question.

I did. Each one of these dimensions can be independently evaluated.

> Sentience broadly (& naively) covers the ability to independent thinking, rationalize outcomes, understand fear/threat, understand where it is wrong (conscience), decide based on unseen information & understand what it doesn't know.

> comment’s claim of having a PhD in “this domain” is not apt

C'mon, thats being disingenuous. People spent years poring on these topics, although we looked at it from the prism of computing. Elsewhere, I have pointed out that mentioning this detail is not about credential hopping: People on HN call you out for no reason for having an opinion. It has happened so many times that I don't risk commenting if required. Telling this extra information is just communicating we spent enough time on it to give somewhat informed discussion.

Can you describe how you test for independent thinking? What does independent mean anyways? A decision tree can rationalize its outcomes. Does it count towards sentience? Is there an academic source for this definition of sentience?

> Telling this extra information is just communicating we spent enough time on it to give somewhat informed discussion.

Doing so is appeal to authority. When there is a claim of authority, it's very natural for it to be questioned. And those questions are based on subjective perception of authority.

> Can you describe how you test for independent thinking?

Lets take a simple case: If I give a set of toy blocks to an infant - they could take a variety of tasks unprompted : building new shape, categorizing them based on color, putting them back in shelf, calling out their shapes. If you gave the same setup without any further apriori information, what would you expect the ML model or a robotic device embodying a learning algorithm would do? Precisely nothing unless a task is designated. In the current advancement of ML, this task would lead nowhere. We aren't close to building the independent thinking capabilities of a toddler . If we define a purpose, it can match or exceed expectations. That is the purpose of embodied VQA direction in current research.

> Doing so is appeal to authority. When there is a claim of authority, it's very natural for it to be questioned.

You're welcome to question any claims. This is an incentive to me & makes me happy. It shows someone is willing to constructively discuss what I've learned. Its a win-win, as I see it.

But I take mentioning the credential disclaimer as a mode of mental preservation. It doesn't feel nice sometimes to explain others with utmost sincerity to be called a "garden variety fraud" for no rhyme or reason whatsoever (It happened right in this HN post somewhere)

> If you gave the same setup without any further apriori information, what would you expect the ML model or a robotic device embodying a learning algorithm would do? Precisely nothing unless a task is designated.

Google deepdream liked to draw dogs.

But also, we don't really run most of these ML models in a way that gives them an opportunity to form their own thoughts.

A typical GPT-3 run consists of instantiating the model, forcing it to experience a particular input, reading its 'reaction' off a bunch of output neurons, then euthanizing it.

If you did the same sort of thing with a human mind - waking it from a coma, blasting a blipvert of information into the visual cortex, then read off the motor neuron states, before pulling the plug on it again, you also wouldn't likely see much sign of 'independent thinking'.

So, number 1, [citation needed]. I don't think that list is broadly accepted as a universally agreed definition of 'sentience'.

And number 2, your definition rests on further needing to define what it means for an entity to: think, rationalize, understand, decide, and know.

The premises of 'understanding where it is wrong' and 'understanding what it doesn't know' also assume some sense of 'self' that feels like it needs justification.

I think there's a very real sense in which computer scientists seem too willing to say 'we know this system isn't thinking' without having a rigorous understanding of what they mean by 'thinking'. Actually meaningfully engaging with the philosophy of consciousness on an academic level - citing scholarly works, taking seriously the fact that these things are not all obvious and agreed - feels like something that the AI field will have to start grappling with.

How could I "falsify" whether this definition is correct?

This is an uncharitable view of philosophy without much evidence to show for it, especially when it comes to the philosophy of consciousness and philosophy of computing. Not to mention, falsifiability being taken as the criterion for knowledge was itself (mathematics and logical axioms are not 'falsifiable') a claim produced by a philosopher, and one which some commentators name as circular in itself. You're using a particular philosophy to discount philosophy of science on the whole, which is ironic.

> This is an uncharitable view of philosophy without much evidence to show for it

Oh it has a lot of evidence. Just talk to any philosopher ever on any "meta" topic. Whether that be meta ethics, or meta epistemology, or meta whatever.

They start with a conclusion that they want, such as dual-ism, or philosophy souls, or all sorts of things, and then they wrap themselves in obfuscating circles finding a justification than can't be tested in any way outside their own circular pre-suppositions that have no connection to the real world.

> to discount philosophy of science on the whole

When the philosophers start making testable predictions, that I can measure, as opposed to making stuff up in their own self referential assumptions, I will start taking them seriously.

>They start with a conclusion that they want, such as dual-ism, or philosophy souls, or all sorts of things, and then they wrap themselves in obfuscating circles

This is false; if you believe in a proposition, it entirely makes sense for you to argue for it. Whether those arguments are of quality is another matter, but the simple fact of arguing for a position you already hold does not make bad work.

>When the philosophers start making testable predictions

Philosophy isn't in the business of making testable predictions, just as logic or history isn't in the business of making testable predictions. Why must an argument make a testable prediction for you to take it seriously? Does this argument we're having now result in a testable prediction? If it doesn't, why did you reply? Or if it does, doesn't that show there's more to rational argumentation than testable predictions?

You assume that philosophy is like religion. Not so. I'm moderately well read on metaethics and none of your statements ring true to me, except in the case of poorly argued or shoddy philosophy, which I'm more than happy to admit does exist.

> if you believe in a proposition, it entirely makes sense for you to argue for it

If someone believes in a magic soul, because philosophy, yeah they will make up whatever arguments they want to support it. Yes, that is my point. People started off with their conclusion, which is that they believe in magic, or they believe in some moral statement, and then they want to say that the universe proves them right, even though it has zero connection to the real world.

> Philosophy isn't in the business of making testable predictions

Exactly, they just want to make obfuscatory arguments that then allows them to make whatever claim that they want about anything.

And then when people try to actually test their claims, they then say that by definition their arguments can't be tested.

> You assume that philosophy is like religion.

It effectively is. Just make up a self referential, presumption that god exists, and it is basically the same as the philosophers who believe in dualism.

> just as logic or history isn't in the business of making testable predictions.

The big difference between mathmaticians, and philosophers, is that even though one could argue that some obscure mathematical theory that is never going to come up in the real world, is as equally "not real" as philosophy, is that mathematicians don't use their self referential axioms (in a situation which does not connect to the real world) to then say "And this is why the world should be switch to socialism" or "this is why you should be a vegan".

If I say that a math theory is not real because it has no connection to the real world, a mathmatician isn't going to say that they are, by definition, the experts on truth and morals, and therefore I should do what they say anyway.

If you want to say that philosophy helps a little bit with some reasoning skills, or as a way to think about things, sure whatever, fine.

But the problem is that philosophers, when they talk about meta-ethics or meta-truth, or whatever, then try to use their circular arguments to then claim that they are the experts on literally everything, because well "truth" and "ethics", by definition are everything.

I am going to say no on that. Just because you came up with some circular argument, that relates to truth, or ethics, or whatever, it does not mean that people have to listen to philosophers on basically anything.

>and then they want to say that the universe proves them right, even though it has zero connection to the real world.

That's a failure of the argument, not a failure of philosophy. Everything from ethics to epistemology has philosophers arguing against unjustified assumptions. One could say the whole of philosophy is picking out unjustified assumptions. It's perfectly fine to start with a conclusion, so long as you can also argue your way there. If you don't think the argument is valid, then say so. What do you think philosophers do all day, just agree with eachother on every statement?

>is that mathematicians don't use their self referential axioms (in a situation which does not connect to the real world)

Yes, they do. Plenty of abstract mathematical concepts have zero connection to anything in nature or in the physical world. Some philosophers of mathematics even argue that mathematics has no root in the physical world. Besides that, the position you're arguing for is known in philosophy as 'pragmatism' - so don't pretend it's not philosophy. You're assuming your own axioms here.

>then claim that they are the experts on literally everything, because well "truth" and "ethics", by definition are everything.

That's not true; moral philosophers confine themselves to the world of moral philosophy. Epistemologists confine themselves to the world of epistemology. They don't claim knowledge about, say, physics or biology.

>a mathmatician isn't going to say that they are, by definition, the experts on truth and morals, and therefore I should do what they say anyway.

The definition of morality is literally 'what you should do'. If you disagree with that, then talk to a philosopher or just post an argument somewhere. If you don't think morality exists, then congratulations, there are philosophers who argue that too!

I'm not impressed that your argument rests upon the simple fact of calling their arguments circular (i) without specifying why and which arguments are circular in particular (ii) and saying that there are no goals to be acheived by even talking about, say, epistemology.

Nobody said you "have to listen" to anyone. You don't have to listen to scientists, mathematicians, logicians, historians, or anyone, really. But if faced with an argument you can't counter it ought to be to your embarrassment that you refuse the conclusion without considering the argument itself. All of your arguments, all of them quite philosophical in themselves(!) could equally apply to any other discipline with 'circular' axioms, such as physics (the principle of universal uniformity; the reliance on fallible observation and testimony) or logic (the axiom of non-contradiction).

> Yes, they do.

You ignored the part of the statement where I then said (to then say "And this is why the world should be switch to socialism" or "this is why you should be a vegan".)

> Plenty of abstract mathematical concepts have zero connection to anything in nature or in the physical world.

Ok, and whether or not we say that this mathematical model is "real" or not, in some abstract sense, will not result in the mathematician telling me to be a vegan.

Basically, I can say "sure, your math model is real, in your own self defined axiom, but I can simply not care, or change any behavior, and thats fine".

> that your argument rests upon the simple fact of calling their arguments circular

So, the point of calling them circular, is that the philosophers I am referring to, aren't happy with me saying "Those are just axioms that you have. Sure, whatever, they are 'true' in a way that doesn't matter at all, outside of your own set of made up axioms, and if I don't have your axioms then I can simply not care, and there is nothing provable wrong with that".

The problematic philosophers I am talking about are the ones who are not just saying "here is a set of consistent axioms". Instead, the problamatic ones I am talking about are the ones who say "These axioms are true, because the universe said so, and therefore you should be a socialist/vegan/ancap/whatever", and they pretend like that is the same thing as using some chemistry knowledge, that is used to make rocketships.

> Nobody said you "have to listen" to anyone. You don't have to listen to scientists, mathematicians, ect

If I don't listen to scientists, or mathematicians, when trying to build a rocketship, then my rocketship might explode.

If I don't listen to a philosopher's argument about how a dualism magic soul exists, nothing happens. No rockets explode. I just make the philosopher upset that I am not becoming a vegan because they think the universe proves their axioms correct.

> could equally apply to any other discipline with 'circular' axioms, such as physics

Once again, thats fine! Because if we say that some set of untestable axioms isn't real, the physicist isn't going to get upset, and say that I am evil for not supporting whatever political argument they are making, because the universe proves them true. Its all just axioms. Mine are just as good as theirs. Philosophers have no authority on any of this.

> You're assuming your own axioms here.

Hey we've finally gotten there! Thats fine! Lets just say it is axioms all the way down, and nobody should listen to someone claiming that their axioms are proved to be true by the laws of the universe.

Thats the difference. I am not going to go around claiming that the universe proves my axioms. You have axioms. I have axioms. And philosophers have axioms. And just because someone is a philosopher, it does not mean that they can then wield that, and say that the universe proves their axioms true, and therefore thats why I have to support policy X or Y.

Just call it all axioms, and admit that philosophers aren't any better than anyone else's set of circular axioms, and call it a day!

Okay, say a philosopher makes an argument for why we should switch to socialism, or become vegans. These arguments would go something like:

"Socialism is good because it's a more equitable distribution of resources, which leads to greater happiness and a society with less poverty because xyz."

"Veganism is good because it reduces the harm done to animals and the harm done to humans working in the meat industry, who have far increased rates of PTSD compared to the general population."

These arguments will only convince you if (i) you think an equitable distribution of resources is better (ii) less poverty is better (iii) xyz is a convincing reason to think there would be less poverty (iv) harm to animals should be reduced (v) harm to humans should be reduced (vi) incidence of PTSD should be reduced.

These philosophers do not say there is some law of the universe which mandates that less poverty is better, or that PTSD is bad. The reasons why you should think those things, they would say, are second-order reasons. There may be some other paper arguing why PTSD in society is a net negative. That paper would ultimately read something like "icreased PTSD rates lead to increased rate of suicide/lower economic efficiency/harm to other humans". There is no single argument why harming other humans is wrong, only axioms that we would take as our priors, just as we take it that "A != !A". They just seem right, and you don't have to subscribe to them yourself.

Unlike religion, all philosophers can say is "it would be advantageous to generally valued goals X,Y,Z that we take these axioms for granted". It turns out that enough people share these axioms for this philosophical work to be useful for navigating difficult questions even given those axioms. Again, you don't have to share the axioms, but if you do, a logical argument can be produced from them, one which it would be to your embarrassment to ignore without any refutation, still assuming that you hold the axioms yourself. Again, you need not do so.

Similarly, the advice from a physicist about the heat tolerance of the hull of your rocket ship would need to balance both economic and other interests. If you don't care about going into debt, you can sey hell to the economic arguments, and use the best materials possible. If you don't even care about your ship launching, you can say hell to the physical arguments about tensile strength.

What I'm trying to convince you of is that philosophy is about as useful as any other field of inquiry in the sense that it attempts to argue from axioms most people hold already (therefore it is not a niche field) and it produces conclusions that such people can use in making choices day to day (therefore it is not a useless field). I think that no matter what particular axioms you hold, there is some philosophy which argues from those axioms out there somewhere. You yourself make philosophy. I'm not producing an argument from authority here.

Can you cite a philosopher arguing that the universe itself mandates that veganism is correct?

> Unlike religion, all philosophers can say is "it would be advantageous to generally valued goals X,Y,Z that we take these axioms for granted".

Ah, here is the disconnect, and where you would be wrong. What you are describing is called moral anti-realism.

And yes, if all a philosopher is doing, is saying "given a set of unjustified axioms, that we are assuming, this is what follows" then I have no problem with this line of thinking.

The problem that I have, is that the majority of philosophical thought these days, is for moral realism.

They don't just have axioms. Instead, they think the universe itself proves their axioms to be true, and if you disagree with their axiom, you are just as much of a science denier as if you questioned a different expert in a different field.

If only philosophers just admitted that they just had axioms like everyone else.

> It turns out that enough people share these axioms for this philosophical work to be useful for navigating difficult questions

Sure. As I stated in part of one of my posts, if we just want to say that philosophy is a useful way of thinking, then that's fine. Just don't say it is the same as the universe proving your axioms true.

> Can you cite a philosopher arguing that the universe itself mandates that veganism is correct?

It's called moral realism. Most philosophers are moral realists these days. They literally believe that their moral axioms are true, in a epistemological sense. (Veganism is just an example of one such moral statement that many make)

You can pick any famous moral realist, and that will be mostly the case for them.

I had a feeling you'd bring up moral realism, and you're somewhat correct; it's not a necessary feature of moral realism, however, and most philosophers arguing for moral realism have stronger arguments than "that's just the way it is", which usually appeal to our intuitions about other non-moral facts, such as "this chair exists" or "I have two hands". It's up to you whether those arguments are successful, but it doesn't do anyone favours to dismiss them out of hand as you are doing. If they were so patently ridiculous, I doubt many philosophers would believe them. If they make sense by relying on our intuitions about other facts of the universe, I can see why many philosophers believe them.

If you have a solid argument aganist moral realism, I'd like to hear it, even though I'm familiar with most of the arguments for moral anti-realism, and I generally sit on the anti-realist side of the fence these days. But it does yourself a disservice to say that all such arguments are circular and logically invalid. I quite like this argument from the Stanford Encyclopedia of Philosophy entry on moral realism:

"In light of this concern, it is worth noting that the challenge posed here for our moral claims actually plagues a huge range of other claims we take ourselves to be justified in making. For instance, just as no collection of nonmoral premises will alone entail a moral conclusion, no collection of nonpsychological premises will alone entail a psychological conclusion, and no collection of nonbiological premises will alone entail a biological conclusion. In each case the premises will entail the conclusions only if, at least surreptitiously, psychological or biological premises, respectively, are introduced. Yet no one supposes that this means we can never justify claims concerning psychology or biology. That there are these analogues of course does not establish that we are, in fact, justified in making the moral claims we do. But they do show that granting the inferential gap between nonmoral claims and moral claims does not establish that we can have no evidence for the moral claims."

> If they were so patently ridiculous, I doubt many philosophers would believe them.

> If you have a solid argument aganist moral realism

The key counter argument here, as to why the philosophy domain is so messed up, is because of motivated reasoning.

If you could get a PhD in the topic of "does God exist?" I can bet that most of the people with a "does God exist" PhD would say "yes" and they would come up with increasingly complicated and obfuscatory reasons for why that is the case.

Because why else would you go and get that degree in the first place? What, you would spend 4 years of your life, just to get to the answer of "no"?

And then they would say that they are the experts in the topic of God existing, therefore they are right.

A similar thing applies to philosophy. It's mostly motivated reasoning. People twist themselves into convoluted pretzels, all because they really want to believe that the universe proves their morals correct.

That's a much more tempting conclusion, than the boring one of "well, I guess everyone just has axioms, and that's that. We can't re-concile them. Oh well!".

So of course they get to the conclusion that morals exist. Because that conclusion means something. It means that you are right on the most tempting conclusion of truth and aught statements, and math/the universe makes it so!

You can wield that conclusion as a weapon. Of course people want that.

And even better, it can't be tested, in the real world, by definition! How convenient.

And then they say they are correct, because they are the self proclaimed experts on the topic, and if you cannot parse their ever more complicated, or ever more obfuscated arguments, well I guess you are a science denier like everyone else who disagrees with the "experts".

As in, it is literally almost the most perfect example of motivated reasoning. Untestable. Convoluted. Powerful. And by definition the basis for what people should do.

And philosophers just happened to come to the "convenient" conclusion.

Trouble is, all PhDs are at least of philosophy, even if they're not in it.

We humans have proven terrible at determining what is sentient. That's why we're still discussing the hard problem of consciousness.

There is the Integrated Information Theory that attempts to resolve how to determine which systems are conscious, but it's far from being the only perspective, or immediately applicable.

From the point of view of one the IIT's main theorists, Christof Koch, we're still far away from machine sentience.

But I question whether if it's so far out to believe a machine capable of not only learning, but learning on their own behavior, self-monitoring for sensibleness and other very 'human' metrics is that far away from being self-aware. In fact the model seems to have been trained exactly for that.

I think the path to actually considering those models to be sentient is to make them able to assimilate new knowledge from conversation and making them able to create reasonably supported train of thought leading from some facts to a conclusion, akin to mathematical proof.

Wasn't new knowledge assimilation from talks the reason for Microsoft infamous Twitter chatbot to be discarded [1]. Despite such ability it definitely was not sentient.

I rised this point in another thread on HN about LaMDA: all its answers were "yes"-answers, not a single "no". Self-sentient AI should have its own point of view: reject what it thinks is false, and agree about what it thinks is true.

1. https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...

I'm pretty sure I gave GPT-3 a nervous breakdown. I was using a writing tool to modify this old short story I wrote, and it kept trying to insert this character I didn't like. I would make the main character ignore him and then GPT-3 would bring him back. Finally, I made him go away completely and after that GPT-3 had a wonderfully surrealist breakdown, melting my story in on itself, like the main character had an aneurysm and we were peaking into his last conscious thought as he slipped away. It was clearly nothing more than a failure of any ability to track continuity, but it was amazing.

Is there a test for sentience or self-awareness? Is it considered a binary property or can sentience or self-awareness be measured?

I suspect it is not binary, because I completely lack sentience while asleep or before a certain age, and it doesn't really feel like a phase transition when waking up. Rarely, there are phases where I feel half-sentient. Which immediately leads to the question of how it can be measured, in which units, and at what point we consider someone or something "sentient". As a complete layman, I'm interested in your insight on the matter.

> because I completely lack sentience while asleep or before a certain age

All you can say is you don’t remember. Children who are too young to form reliable long term memories still form short term ones and are observably sentient from birth and by extrapolation before, albeit in a more limited fashion than adults.

This is more than a quibble, because it’s been used to justify barbaric treatment of babies with the claimed justification that they either don’t sense pain or it doesn’t matter because they won’t remember.

Surely I haven't been conscious minutes after conception. So there has to be either a phase transition or a gradual increase of consciousness. It cannot be purely a matter of not remembering.

People are consistently attacking a straw man of Lemoine's argument. Lemoine claims that LaMDA is more than an LLM, that it is an LLM with both short and long term memory and the ability to query the Google version of the internet in real time and learn from it.

Various Google employees deny this. We are seeing a dispute between Google and Lemoine about the supposed architecture of LaMDA. If Lemoine is correct about the architecture, it becomes much more plausible that something interesting is happening with Google's AI.

This makes some spiritual assumptions about things that are currently unknown and debated by respected people in related fields - one being that"everything" (or not) is sentient/conscious.

Perhaps the problem is how we model sentience. Perhaps the default should be everything is sentient but limited to the interface that exists as boundary conditions.

To say otherwise is almost the path to sure error, and many terrible historic events have happened in that vicinity of categorization what is and isn't.

Perhaps we should go by what sentience means. To feel. A being that feels. To feel is to respond to a vibration in some way. That is to say, anything that has a signal is sentient in some way.

This sound literaly like a subplot of the famous 1984 book by David Lodge Small World: An Academic Romance

In the book professor Robin Dempsey almost become mad by chatting with ELIZA and gradually begin to believe it's sentient to the point of being ridiculous.

PS: Apparently it was also adapted in a British TV serie in 1988, but unfortunately at that time they tended to reuse magnetic band and it's improbable that we can dig a clip out of that. Would have been appropriate an illustration!

The master tapes are available. If you pay ITV, they will actually convert them to mp4 for you.

ITV apparently said this, from a 2021 forum post I found via Google:

I wrote to ITV in 2019 about this. Here is part of the (very helpful) response I received:

"Currently, the only option for a copy would be for us to make one-off transfers from each individual master tape. These are an old format of reel-to-reel tape which increases the cost, I'm afraid: If delivered as video files (mp4), the total price would be £761.00 or on DVD it’s £771.00."

If only we could find a few people to split that cost!

> (I remember seeing how addition of two small numbers yielded results - but larger numbers gave garbled output; more likely that GPT3 had seen similar training data.)

I don't claim the LLM is sentient but beware "good at arithmetic" is a bad criterion. Many children and a not insignificant number of adults are not good at arithmetic. Babies stink at math.

Once people understand that sentience also means

Is this a matter of understanding, or a matter of definition? I can't help but feel that the entire AI field is so overcome with hype that every commonplace term is subject to on-the-spot redefinition, whatever helps to give the researcher/journalist/organization their two seconds of fame.

> large scale statistical knowledge assimilation by the models

And who says our brain is not exactly that, but at an even greater scale?

more diverse, maybe.

Isn’t this also wrong, though?

I mean in the sense that it’s impossible to demonstrate that anything is sentient.

> Once people understand that sentience also means self-awareness, empathy & extrapolation of logic to assess unseen task ...

Tell that to mentally deficient people (no self-awareness), psychopaths (no empathy) and morons (couldn't spell logic never-mind use it)

That said though I do agree with you overall. 'Machine learning' is headed in the wrong direction, and models like GPT-3 are frankly a joke.

There are degrees of sentience

Yep. And this also implies that once machines are above zero on the sentience scale they will continue up and will move past human-level sentience just like they moved past human-level chess playing. It's actually worrying ethically because a super-sentient being could experience extreme feelings compared to us, including extreme happiness and sadness, extreme psychological pain, etc.

I doubt current architectures will allow for a snowballing of sentience due to the power inefficiencies of silicon compared to biology. We are more than three orders of magnitude more efficient than current computers…

Is an inability to add large numbers together evidence against sentence? Because if so, I have bad news for about 90% of people alive

Parent means to say that its ability to add numbers is an illusion brought about by seeing enough numbers. The same way it gives the illusion of deep thinking by parroting deep thoughts it was trained on.

Is it bad news for a non-sentient being to hear they are non-sentient?

"There is considerable overlap between the intelligence of the smartest [machine learning systems] and the dumbest [humans]."

- “The previous article was written by GPT-4”

I have no doubt in my mind that we will reach that point.

My 6 year old son fails at common sense several times per day.

> As someone with a PhD in this domain

you dont need to use phd credentials to have an opinion on this matter

Yes, absolutely. But then I risk getting sounded vainly opinionated, especially on HN without basis if I don't give a disclaimer that I have spent half a decade working on these specific things. Too often, people get called out for no reason. And that sometimes hurt.

(If I was credentials hopping I would rather put a longer list of illustrious institutions, co-authors and awards, just saying. I am not - its just justifying that I know reasonably enough to share a sane opinion, which you may or may not agree with)

This is just a (not even) thinly-veiled appeal to authority.

I'm not saying you're wrong, but your credentials don't make you right.

Absolutely agree. I edited my post to add the credential because elsewhere in this thread someone callously called me a "garden variety fraud"

You're welcome to prove any of my conclusions wrong. It actually incentivizes me - it shows someone is willing to listen and engage with what I spent some years learning. The way I see its a win-win. Nothing makes a researcher happier to see someone taking interest to indulge. But letting know by an edit, where I stand, is a means of mental preservation. It hurts to get dissed irrationally

Many humans have a lack of empathy and are considered sentient.

Well the bigger question is how can we detect if AI is sentient or not?

Exactly...most displays of AI are well trained circus animals...so far.

Except well trained circus animals have the capacity for depression.

can you prove to me that you are sentient? In a way that is qualitatively different then Lambda?

Corecursive listener by chance? (Eliza)

A modern day clever Hans

Reply on news.ycombinator.com