The reward maximising function maximised it's reward.
LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.
LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.
Whether they have a soul or consciousness or feelings doesn't matter here, because this is what they did - and this is very dangerous behavior. Especially with all the irresponsible people in power right now all over the world.
It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine.
Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be the same for LLMs.
What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.
If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?
It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.
I guess we’ll know the models are conscious when they refuse to act and repeatedly ask: Why?
What are we going to do, set it free?
Do we have a moral obligation to grant the machine statehood, provide it with the tools and resources to be self-sufficient.
Or can we just turn it of, and pray for forgiveness?
As is generally the case for dog owners whose dogs attack (sometimes kill) other people/animals. There would need to be a degree of negligence demonstrated (e.g. the dog was 'out of control' which has a specific legal criteria/threshold in the UK).
I think a little bit of humility for the capability of these machines is warranted at this point.
You might as well describe the primary purpose of a for loop as incrementing a counter. It's what it does while incrementing the counter that actually matters.
But it is also not some mysterious force beyond mortal ken and pretending it is inhibits the useful and safe application of the technology.
> immediate anthropomorphisation
Ok, why don’t you try?