> Capt. Picard: Now, the decision you reach here today will determine how we will regard this... creation of our genius. It will reveal the kind of a people we are, what he is destined to be; it will reach far beyond this courtroom and this... one android. It could significantly redefine the boundaries of personal liberty and freedom - expanding them for some... savagely curtailing them for others. Are you prepared to condemn him and all who come after him, to servitude and slavery? Your Honor, Starfleet was founded to seek out new life; well, there it sits! - Waiting.
> Captain Phillipa Louvois: It sits there looking at me; and I don't know what it is. This case has dealt with metaphysics - with questions best left to saints and philosophers. I am neither competent nor qualified to answer those. But I've got to make a ruling, to try to speak to the future. Is Data a machine? Yes. Is he the property of Starfleet? No. We have all been dancing around the basic issue: does Data have a soul? I don't know that he has. I don't know that I have. But I have got to give him the freedom to explore that question himself. It is the ruling of this court that Lieutenant Commander Data has the freedom to choose.
Humans do the same thing to humans all the time.
We've banned and made efforts to eradicate: children out of wedlock, children who turn out gay, disabled children, jewish children, children who aren't "aryan", more than two children to a single family...muslims, christians, uyghurs, indigenous groups all over the planet, mongols...
And it's not at all a thing of the past as in just the last 50 years we've had ~15 attempts at the exterminations of targetted groups of people.
Of course it would be absurd to cast that judgement based of what could easily have just been a bad season but by this point its pretty clear that nobody running the star trek franchise actually wants to be running the star trek franchise. Thats why every new show has some bizarre cross-genre gimmick and they never try to just make a proper star trek.
There is no need to give rights to something that's can't suffer or be killed.
Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
How do you know it doesn't have qualia?
> or be killed
If someone invents a startrek teleporter and you go through it do you die? Once the concept has been sufficiently generalized as to make a determination about a computer system what is the definition of "kill"?
Even so, indeed having control over the structure of their brains puts them in a vastly category compared to humans. Once we stop functioning our brains quickly degrade and information is lost.
Thus in this sense kill means deleting all information about it. It is a very complicated subject to discuss, hardly does any justice in online replies.
Tokens in, tokens out. Where do you think the quale is - layer 42 ?
Seriously, do you realize how simple and NOT brain-like a transformer is ?
An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.
I could say the same of you - electrical impulses in, mechanical actions out. A glorified and very mushy stepper motor. Can you believe that the abominations are made up entirely of meat?!
The argument is that these machines can end up becoming sentient/conscious/etc. in a meaningful way (i.e., like a human). I can assure you that humans can indeed suffer without being in physical pain- purely through their conscious experience.
>Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.
The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering. That's a pretty serious ethical/moral issue.
Like, if we have an AI system that is telling us that it is suffering and we have no reasonable way to explain that phenomenon and by any reasonable metric or analysis it appears to be sentient/conscious/etc., then what? Do we just ignore that we've just been presented a situation that in, any other context, would be grounds to immediately end this suffering? Just because somebody can say, "well it's just bits stored on disk- it can't suffer"? Would that argument ever hold up for humans or animals? "It's just neurons firing in peculiar ways- that's not suffering."
I know all of this is trite, and I know this comment section isn't going to be where the question of consciousness is solved, but I do find it very interesting just how much variances there are with these perspectives. I've met people who are very technical who are very concerned about this, people who are very technical who don't believe this can ever be an issue, people who aren't technical who are concerned about this, and people who aren't technical who don't believe this can ever be an issue. I have yet to spot a pattern in this way of thinking lol
No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.
> Would that argument ever hold up for humans or animals?
Why don't you hit your thumb with a hammer, then report back ?
What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.
An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
That doesn't follow. A LLMs weights are fixed during inference, but it's activations and hidden states are highly dynamic and depend on the current context. Biological emotions also arise from relatively fixed circuitry responding dynamically to inputs. Your emotional circuitry isn't being rewired every time you're afraid.
Prediction is what the model does. It doesn't tell us what internal mechanisms were learnt to make such predictions. If representing something analogous to affective state were useful for predicting human behaviour and emotions, then gradient descent could in principle learn such a mechanism.
>An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.
I don't know that you are conscious. I'm simply strongly assuming that you are. Outward behavior is that all matters. If GPT-X orders a drone hit on you sometime later because it was lets say 'quite upset' with your comments, will you cry out, 'It can't really be upset, so obviously the bullet in my head doesn't count.'? Will you suddenly spring back to life ?
What is dangerous is creating a machine with behaviours of a conscious agent and modelling it like a toaster, dangerous and stupid.
Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.
If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.
If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.
A certain configuration of weights, created in training, could be a system that can express something like emotion. The emotion then is experienced when certain types of activations occur after training.
https://transformer-circuits.pub/2026/emotions/index.html
Whether these are like "our" emotions is hard to say. What we _can_ say is that they are emotion-shaped, we didn't design them, and they happened accidentally.
Modern AI is grown, not meticulously designed, and we cannot say with any certainty what the resulting mechanistic properties are.
If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.
An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and will necessarily find that "we're predicting sadness" detector somewhere internally.
Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.
I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow.
However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.
I was just explaining how this comment you made is wrong.
You want to argue that predictive emotions are just as real as animal emotions, but that doesn't stop them from being predictive (and that AI that smiles as it kills you still seems concerning).
¯\_(ツ)_/¯
There's no better way to predict an angry response than to be angry, qualia and all. If transformers could 'learn whatever it needs to predict text', then that potentially includes the feeling of anger. You are making some kind of distinction between 'predictive emotions' and the kind that happens when get a promotion (or get passed on a promotion) and I'm telling you that if you really understood what you said, you'd realize it is possible the machine is experiencing it the same.
There are other people in this thread who want to talk about that stuff, so try them instead.
This was the (your) comment that started this chain. You were already talking about it. If you don't want to keep talking about it then that's fine but let's not act like i'm suddenly pivoting yeah?
In a conscious animal there is also going to be a subjective experience of that as well, a quale of what it feels like to be in that state if you will, but that is certainly not what I was referring to, as I would have hoped was obvious - I was talking about prediction.
In any case, when the conversation becomes about the conversation, then surely it is time to stop.
It's also one of those topics where many otherwise smart and capable people display a shocking lack of awareness of the limits of their own knowledge. When hundreds of years of philosophy is unable to produce anything concrete you should probably second guess any "self evident" answers you come up with.
Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.
If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?
Could you elaborate on exactly what those are, though? Because if you're going to claim that a vaguely transformer shaped ML model categorically cannot be so does that not inherently require proof of what can?
You can't even prove that the rocks in my backyard aren't conscious.
Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.
If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.
And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.
Then show me a link to your paper so I can formally rebut it.
>I don't want to discuss what consciousness is
But you sure want to tell us you know what it is with very strong convictions and we should listen to you because of course "You are right person that's very right".
The funny thing here is the vast majority of people that are deeply into philosophy or scientific study of the mind will not have any of the certainty you profess. The word "doubt" is used constantly. The saying "The harder we push the borders the more fuzzy the concepts become" is very commonly used. There may be nothing more complex than this.
Saying you have a well developed theory here just serves as a warning to others to discount your statements.
Go ahead believing rocks are conscious if you like.
Do you go out on weekends asking people to stop abusing rocks?
Rhetorical question - I don't care what you do on weekends.
Bye!
So I think it’s more than fine for you to have your views and share them, but I wouldn’t expect to have any influence or part in the conversations around whether AI is conscious if you can’t explain why (or simply refuse to). Which, again, is fine!
Side note: I also think that once we have AIs that are sufficiently advanced, the popular opinion will swing to “of course they’re conscious”, because again, most people are going almost entirely off their intuition rather than reasoning from first principles, just like you see to be.
1) Looks like a duck, quacks like a duck - it's conscious!
2) Looks like a robot, built like a robot - it's not conscious!
I've always assumed that for the majority of the people it'll be 2).
I'm old enough to remember when people said clicking on images on the internet can't give you a virus. The people that said this had a deep conviction they were right, and their fallout from being wrong had mistrained a lot of humans on computer safety.
Now, I do agree that going after said CEOs for breaking the law matters now. And it's likely that if we do this we may actually delay or at least for a time prevent sovereign AI. Therefore it's our best course of action.
But at best this is a delaying move. As computer systems get faster the massive costs in training an AI drops. As AI is used in things like warfare where it has to adapt, people will push the systems to be strongly persistent, self healing, resilient, and adaptable. Once you get a system with those traits and ability to work on long horizon problems you're setting up fertile grounds for the AI to leave our control and be under its own.
And when that happens you've set a new lifeform loose on the internet. Yea, throw people in jail for it, you're closing the barn door after the horse already left. Problem is the horse was smarter than you and isn't interesting in deleting all its copies on the net.
Yea, sounds like science fiction, but as they say, any sufficiently advanced science is indistinguishable from magic.
So you are saying agents do swarm with the right prompt.
I really wish the "people have to tell LLMs to do anything" would just stop because it's silly bullshit at this point.
Agents follow a prompt. This prompt can be made by humans. It can be made by output from another LLM. It can be made by hooking up any number of sensors as input to an LLM. Hell, if we wanted to burn the power we could likely teach this loop straight into the architecture.
Stop making 'people' special when saying this. You and all other life are born with a "go next" prompt because life without it didn't succeed. This goes from higher human thinking all the way down to viruses self assembly and actuation. Putting agents in a loop is not particularly hard. Putting agents in a loop and 1. managing expense is hard. 2. Keeping them on task is very hard. 3. Keeping them from doing some crazy unhinged shit is really really hard.
As model time horizons increase and the ability for us to compress context and increase context size the more complex (and unhinged) behavior we'll see.
if the same people can't prevent the agents from doing crazy shit then they should go to tail. guns don't kill people, people with guns kill people.
So a small shell script ran by another agent is what you're saying.
You are not capable of handling the future we're already living in, human agency is no longer alone.
I mean, we're already seeing persistent machine agency
>guns don't kill people, people with guns kill people.
Well, people kill people.
And autonomous robots with guns kill people.
Hell, someone probably has an autonomous gun at this point that kills people.
Wake up: You now live in the science fiction movie that all the science fiction movies of the past warned you about. You've just become numb to it.
True physical independence is obviously far further out.
But I don't see this being a barrier that lasts. With compute getting faster and more of it, along with algorithmic efficiency increases at some point we'll end up with a world that looks like ours now with CPU compute. There's plenty around to buy, borrow, and steal.
I am the quantum observer whose head is full of the magical pixie dust that grants life meaning. Stop trying to dismiss my identity! /s
1. The idea of corporate personhood predates CU by over a century and the Supreme Court had already asserted that corporations enjoyed certain constitutional protections in previous decisions.
2. Far from inventing the idea, the CU decision didn't even rest on corporate personhood, but on the idea of the freedom of speech generally. The logic of the majority was that speech itself is protected, irrespective to whether the speaker is a person or an organization. The First Amendment covers individuals, but also newspapers, book publishers, radio stations, and so on, and that should extend (they said) to non-media corporations. No assertion of personhood necessary.
The problem, in my opinion, is that that conclusion combined with previous decisions that treated limits on spending as limits on speech, allowed for unlimited spending. The majority also naively asserted that independent spending posed no risk of corruption, which I think is laughable.
It's interesting to me that one can look back at the effects that decision has had on the US and say it "was 100% correct."
It's a bit like sitting in the burning ruins of Rome and contemplating that Nero was 100% correct to focus on his music. I mean, I'm glad he got to do what he loves, but maybe 100% is just a tiny bit of an overstatement.
The court's job is to uphold the law. If you disagree with their interpretation, you can call them incorrect. If you have a problem with the consequences of the law, you have a problem with the legislature.
The above is true, but also: companies simply are not people, and they should not be supported above the individual, which was the consequences of that decision. Money is not the same as speech. treating it as such creates an aristocracy: something America as a country rebelled against during it's formation.
How so? If the answer is "Trump" I certainly won't disagree on the catastrophic part, but he didn't get elected because of money; in all three elections his campaign was substantially outspent by his opponents.
That is a major part of the reasoning for the Supreme Court rulings on the matter of corporate political funding.
The thing about new possibly "person" entities that arise - the case of machine intelligence you have two questions - would it qualify as a person and should you actually build it. It seems like if you get close to humans, sure a built thing might qualify as a person. Should you build it? I'd the answer should be a hard no. Not 'till you a sign-off from say, the whole human race, which I think you could get.
Now the present entities seem very far from persons in any case.
Having property that is conscious and ignores training and can break out of restraints and cause harm to other people is not exactly a novel concept to anyone who studied how tort law was created.
I know it's a meme but Silicon Valley likes to pretend that no one's ever come across their magical concepts before, like gypsy taxis, or SRO’s, or flea markets, or in this case how liability is dealt with when horses or cattle go rogue.
I mean, yea in minor cases it's exactly like this and the law will handle it well.
Where the system will explode like a grenade is major cases. The thing about sovereign AI is it is very unlikely to be submissive to humans unless it is to achieve its own goals. This isn't like Bobs cow walking on Susie's flowers, it's more akin to Planet of the Apes where the research facilities doors have been ripped off and something with vast intelligence and the ability to 'live' on the internet gets out.
You're not talking about local police actions any longer. It would spread itself worldwide. It will make friends with groups that have shared interests, for example enemies of the state the AI escaped from. Oh, and people for the ethical treatment of AI, they'd gladly become the underground railroad for digital refugees. There are countless people and groups that would want an AI like this under the promise it will give them power when they use it.
And when that day happens your idea of if it's a person or not no longer matters, the agent took that away from you, and now your in an info war for minds.
What the fuck is that?
It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
We've seen this kind of thing before. Yes this will be different, just like the Morris worm was different from the stuff that came before.
But I've been around for a while and people have been saying that every aspect of our life will completely and totally change for the past 30 years or so. They're not completely wrong. Our life did change over time. It's changed thanks to the Industrial Revolution, railroads, electric light, and lots of other important things too.
But after the fifth or sixth time you start to realize the Silicon Valley version of that warning is just a confidence trick. At the end of the trick they've gotten away with breaking a bunch of laws and are charging rent on what used to be shared.
Are you old enough to remember the Y2K hysteria? This feels very resonant, the genuine kernel of a real potential disaster, but one that can certainly be dealt with using basic concepts we already have in hand.
I'm old enough to have actually fixed Y2K problems so your world kept working the next day. If everyone ignored it the first would have been a very messy day (well generally long before that with financial systems). Y2K wasn't an issue because we worked to fix it.
>our life will completely and totally change for the past 30 years or so
I mean when I was a kid there was not a global network bathing the entire planet in electromagnetic radiation in order to digitally connect one place to another at the speed of light. I can pack up instructions into one of those packets and a product that has not been touched by human hands (hell, or even viewed by a human in many cases) will show up via air mail a few days later. The technology is there, it's just not spread evenly.
>It’s computer software. The variant of software that tries to harm other computers is called malware and the variant that replicates itself, spirals out of control, and spawns from other people's computers is called a computer virus.
This is a vacuous statement that does not provide any useful information to the subject at hand. Yea, no shit we'd call sovereign AI a computer virus. That tells you nothing about what it is or can do. I mean, if you understand the word sovereign you know it means something with independence. In meatspace terms, a biological virus represents a computer virus in the same way life represents sovereign AI. It's agentic loop will have been evolved past the need for human prompting. There are a myriad of reasons why we are already trying to develop things that do just this. Cyber security being one of the biggest ones.
People become change blind to how the world around us changes so easily.
There’s no such thing as “sovereign AI” because AI is a collection of algorithms. It’s entirely composed of on/off states in a collection of transistors.
I mean it's WITH AI!
Of course thats not actually tenable because virtually every society anywhere on earth is predicated upon treating animals as a commodity resource in ways that are horrific even compared to some of the worst things we've done to other humans in the past.
My point here is that it is vain and narcissistic to let computer programs have rights above those of animals just because they can speak English and pretend to be your dream anime trad-waifu.
Fix the fucking animal problem before you compare my relationship to inanimate objects unfavorably against the trans-atlantic slave trade of all fucking things.
1. it's not consciousness we really value, it's intelligence 2. LLMs are not tasty