Judge: Did you hear about the Irishman who found a magic lamp? When he rubbed it a genie appeared and granted him three wishes. “I’ll have a pint of Guiness!” the Irishman replied and immediately it appeared. The Irishman eagerly set to sipping and then gulping, but the level of Guiness in the glass was always magically restored. After a while the genie became impatient. “Well, what about your second wish?” he asked. Replied the Irishman between gulps, “Oh well, I guess I’ll have another one of these.”
CHINESE ROOM: Very funny. No, I hadn’t heard it– but you know I find ethnic jokes in bad taste. I laughed in spite of myself, but really, I think you should find other topics for us to discuss.
J: Fair enough but I told you the joke because I want you to explain it to me.
CR: Boring! You should never explain jokes.
J: Nevertheless, this is my test question. Can you explain to me how and why the joke “works”?
CR: If you insist. You see, it depends on the assumption that the magically refilling glass will go on refilling forever, so the Irishman has all the stout he can ever drink. So he hardly has a reason for wanting a duplicate but he is so stupid (that’s the part I object to) or so besotted by the alcohol that he doesn’t recognize this, and so, unthinkingly endorsing his delight with his first wish come true, he asks for seconds. These background assumptions aren’t true, of course, but just part of the ambient lore of joke-telling, in which we suspend our disbelief in magic and so forth. By the way we could imagine a somewhat labored continuation in which the Irishman turned out to be “right” in his second wish after all, perhaps he’s planning to throw a big party and one glass won’t refill fast enough to satisfy all his thirsty guests (and it’s no use saving it up in advance– we all know how stale stout loses its taste). We tend not to think of such complications which is part of the explanation of why jokes work. Is that enough?
Dennett: "The fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinary supple, sophisticated, and multilayered system, brimming with “world knowledge” and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, and much, much more…. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all."
[1] Turing: "Are there imaginable digital computers which would do well in the imitation game?"
My level of immersion usually is quite high if I like a game. Even today. Despite crud graphics and AI behavior. I'll use an imagination to tell the missing part of the story or give NPCs goals and desires. Simply because if feels better. By analogy, I would not kill innocent because that feels bad to me. So, if the day will come when all this missing part will get filled by brilliant AI I am not going to kill it either.
Probably the reason that people have no quarrels with killing everyone in a game is that it's very difficult to convince player that NPCs have a feelings. Even in modern AAA titles when you start shooting in the middle of the city it looks more ludicrous than real. People are frightened, sure, but very often they make really fake poses, run randomly with hands up, or do some other stupid things. It's hardly convincing, so shooting them is easy and players have no remorse about it. You can easily see they are fake. But should they behave really human-like, should you see their pain... would you still pull the trigger?
There wouldn't be one AI that felt pain per NPC, it would be a general orchestrator deciding which NPC to make cry and how they should act.
Those in the dream might have arguably better AI than any video game character could hope to have and be as realistic as possible. But they're also not real. They don't matter and simply cannot matter.
Same goes for virtual acts towards real people's avatars. If they're within the rules of the game and what the creators of it intended, then they're not against the law. Because you haven't really done anything to them.
Otherwise you may as well start saying it's wrong to kill players in a MMORPG setting or what not.
Then again, I'm also someone who thinks fictional worlds should be treated as if they were completely seperate universes to the 'real' one in general. Murder, theft, assault, terrorism... right down to copyright infringement shouldn't be banned in a fictional world unless that world's setup explicitly sets laws against those things.
We have to keep fiction and reality seperate.
For the time being I think there will always be a clear divide between biological life (as we know it currently), and "virtual" or machine life, as we create it - I think these ethical questions will come in to play when the two become intermixed enough to not be distinguishable one way or another, which I think is still a far way off.
The reason I say the above is just based on our history as a species, and our current treatment of biological life - I think "virtual" life will likely be treated as 2nd class compared to biological life, at least at the beginning.
I am seeing it completely opposite. Assuming they gain a self awareness, they would know nothing about server they are living on. They wouldn't know a thing about our world or about the player. They'd think that their world is the one and, if any, will see other servers or our world as some sort of "heaven" or another dimension. Would you know if our universe would run on some server?
That is, unless someone told them that they have been created for a sake of entertainment and will die as soon as player is bored enough to uninstall the game. Which would probably make them behave completely different than expected, since they would know they aren't "real". Not to mention that it would simply be a cruel thing to do, considering above mentioned assumption.
Moral quandary solved!
I mean, I could certainly imagine myself having a good time pretending to be the bad guys who fight the player character.
This is a circular argument. Whether a sentient NPC is, in fact, still fiction is the entire question.
The story in the game is fictional, but at some point the medium bottoms out on something with physical reality. The risk is of the medium of the story having rights. The nearest real-world analogue would be if you were trying to put on a realistic Hamlet, so you get some clone people to play the people who die and then you actually kill them. Is that murder? That they died in the course of producing entertainment.