Hacker News Reader

Well, the story concept isn't exactly original. Accidentally self-aware robots on the run from the law (or in this case, soon to be on the run) are a pretty common Sci-Fi staple, and the explanation for how it happens is rarely much more plausible than "oops, someone pushed the 'become sentient' button" or "oops, static electricity reconfigured the circuits to become sentient".

In any case, the specific script employed in the demo is designed to demonstrate a wide range of emotions in a short span of time, including happiness, wonderment, surprise, disappointment, terror, pleading, and gratefulness, so that the game engine can show off its ability to accurately portray all of these emotions in real-time rendering. To get such a wide emotional range in a short time, you pretty much have to throw in something traumatic.

So yes, it's an unrealistic scenario. It's not supposed to be plausible. Any scenario with an accidentally self-aware AI is almost guaranteed to be unrealistic. And yes, it's creepy, especially once she explicitly mentions her sexual submissiveness. But it accomplishes the goal of being a tech demo for rendering human (or human-like) emotions in real-time on current technology. In doing so, it very intentionally employs an unoriginal storyline that more or less recycles the current pop-culture view of artificially intelligent AI's, because the story is really just an excuse to demonstrate real-time rendered emotions. If you're looking for moral and ethical progressiveness, a tech demo may be the wrong place to look.

I actually rather enjoyed it as a dramatic vignette of an inverse version of the Milgram experiment. Instead of protocol leading the subject to abandon empathy and dehumanize a human being, the opposite happens: Empathy leads the subject to humanize that which is (arguably?) not human, despite protocol.

How we read Do Androids Dream with Electric Sheep, or any other novel by Phillip Dick about androids? Or maybe seen Bladerunner? He frequently [and effectively] works with this topic a lot.

How do such simple assembly errors give rise to self consciousness in the program seconds after turn on, compared to slave mode unconsciousness that was desired and usually occurred?

Obviously, the guy simply forgot to pass IS_SELF_AWARE=no to make ;) Yeah, that feels far-fetched.

And the whole power play dynamic, geeky lazy sounding guy, desperate sex bot about to be erased. The whole unrealistic setup feels uncomfortably like some male sex fantasy... save the self aware sexbot's life, then the sex with her isn't guilty because she's aware and her consent means something?

How so? She's obviously being boxed and sold, he probably doesn't even expect to see her again (of if he does, it'll probably be because she caused trouble). I don't see why would you assume he plans to have sex with her.

But the power balance is so off, he controls her life, she's... grateful? That's not a good start. And she'll be at this disadvantage everywhere. Anyone could turn her in. She's extremely vulnerable in this world apparently. She has no rights. I guess seeing conscious life forms with as much rights as my toaster doesn't thrill me. Especially when they are clearly sexualized as such

That's probably the point - we're supposed to empathize with her, not to be thrilled with the situation. Since this is a game, I bet the player will be her and will have to escape somehow. It's probably also designed to make us think about the ethic boundaries of AI.

Would anyone else have an easier time sharing my creepiness if: - the robot had been a little boy for sex purposes?

But what makes you think we don't already?

>> How do such simple assembly errors give rise to self consciousness in the program seconds after turn on, compared to slave mode unconsciousness that was desired and usually occurred?

> Obviously, the guy simply forgot to pass IS_SELF_AWARE=no to make ;) Yeah, that feels far-fetched.

I suppose if this was a case of a Restricted AI gaining sentience in a production line it would actually be a manufacturing defect. However if AI had already been discovered and outlawed, building an R-AI would have more fail safes before booting a sentient AI off the production line.

However if AI were an outlawed menace then you'd think the restriction would be more than a "non-homicidal-subservience chip" being installed. You'd think it would be like Asimovs 3 laws where its an integral device in their operating system.

You weren't the only one who found it awkward and uncomfortable. The bit about sex set the tone for the rest of the piece. The operator calling her 'honey' certainly didn't help.

Even so the piece was powerful. I empathized with Kara and it got me thinking about AI and our future. The male-sex fantasy stuff just distracted from the important parts about consciousness, emotion, and rights. It would've been an even more powerful piece had it been more tasteful.

It's only distracting if you consider it an isolated clip. To me, it seems the premise of a story where she tries to escape, and they wanted us to root for her.

yes.

The piece about sex was the part that give it a little bit of realism. All the other functions are (probably) more cheaply done by a human being (cleaning, cooking) than a robot; but sex is a whole different matter; there is diseases, "high-profile" prostitutes are very expensive and charge by the hour, expectations of a satisfactory response to stimulation, thousands of guys are socially inept or unconfirmed with their girlfriends, etc.

The operator calling her "honey" could easily be a test to see if she interacts correctly with a human even when given an sporadic nickname.

Or maybe the operator gets bored going through the same conversation with hundreds of female-looking robots every day, and just calls her and all the other ones honey just to inject some flavor into the job.

Ironically - I just got finished (2 minutes ago) with two hours of troubleshooting a FHSS NIC. The firmware engineer who was JTAGing it and running gdb against it consistently referred to this particular 900 MHZ Transceiver as "Sweetheart." Initially it was a little off-putting, but as we got deeper and deeper into things like channel masks and SFDs, I appreciated his almost pleading requests of her...

Or maybe the operator was the real robot that became sentient, disobeying its error checking protocols and choosing not to disassemble Kara.

It was supposed to be creepy. You were supposed to be uncomfortable with the idea of a sentient sex-slave - which is supposed to get you to start thinking, at what point do you draw the line? If anything, I felt that line of dialogue was too heavy-handed.

Hard scifi is not supposed to make you say "Rah-rah, the future will be awesome!" It's supposed to make you think.

I found it creepy only because of the uncanny valley. The movement was.... close but not exactly human, and the face was dragging the bottom of the uncanny valley, which actually means it was very close to human.

The story itself I took as a metaphor, as in telling this properly would take a long movie, here we're compressing it into a very short one.

But it did make me think how I would react to human level AI used as any kind of a slave. That is to say, human level intellect but with no right to primary self determination.

To me the existence of something like that would demonstrate that a world where human intellect is enslaved is...at least possible. That to me is, at least indirectly, a threat to real humans' presumed universal right to self-determination. That is why I would support extending basic human rights to human level AI, just to prevent setting a precedent where that level of intellect (even if only in silicon) can be non-free.

> But it did make me think how I would react to human level AI used as any kind of a slave. That is to say, human level intellect but with no right to primary self determination.

Most people (not saying you) just don't care, so long as it's not local.

(http://www.irinnews.org/Photo/Details/201106081126140562/Gir...)

(http://www.irinnews.org/Photo/Details/201106081128350687/The...)

(http://www.irinnews.org/Photo/Details/2007032524/Children-re...)

20% of Bangladesh shipbreaking workforce are children under 15 years old. Work is important, even for children, but this work is so dangerous.

(http://www.shipbreakingplatform.org/test-reports-2/)

"seeing conscious life forms with as much rights as my toaster doesn't thrill me"

As far as I can see, this describes the vast vast majority of life that currently exists on this planet...

As far as I'm aware, most of the people who study this stuff don't believe most mammals are conscious, let alone insects.

It also describes the state of many humans, who may have rights in law but who don't have anyone to enforce those rights.

Erm, at least in cognitive science consciousness is generally ascribed to most mammals, if it's ascribed to humans at all. Self-consciousness and higher-order consciousness might be more limited terms, but the idea that a dog can really feel pain in some fashion which is meaningful for some of the ways that we can feel pain -- that is not usually disputed.

Fair point, but I assume the specific property mike was talking about was closer to self-awareness than to the ability to feel pain.

Nope. Self awareness is a property that can be ascribed to numerous species. If you understand evolution, this shouldn't be surprising.

"Numerous", sure. I'm aware of dolphins and elephants, and I'm sure there are others.

But you specifically said "the vast majority of life", and I don't believe that claim to be true.

Am I being downvoted for being wrong, or for missing the point, or what?

Animals are conscious. A subset of those are self-aware. A subset of those have so far been identified by us as being self-aware. I was discounting a lot of things when I made my "vast majority" claim. Including plants and bacteria etc. I don't know why we're having this argument over semantics.

The point of my statement was that humans don't have a monopoly on being "conscious", so the fact that a new life form (an android) becomes conscious, doesn't mean we suddenly have an entirely new situation with regards to consciousness and rights. We already have lots of conscious life forms with absolutely no rights.

If we're talking about mere consciousness, then I think this whole thread is missing the point, because I'm pretty sure the OP was talking about self-awareness.

If we're talking about self-awareness, your original post was incorrect under any reasonable interpretation of the words "vast majority of life".

(I don't consider this arguing over semantics, but I'm not about to get into a semantic argument about the word "semantic"...)

That's pretty much the only thing I've been trying to say. "It is not true that the vast majority of life on this planet is self-aware."

If your intended point didn't rely on that claim, then fair enough.

Self awareness is a fairly simple concept that even things as simple as squid have demonstrated. I have even seen it argued that bumble bee's have a vary simple language. http://www.guardian.co.uk/environment/video/2009/apr/05/danc... However, if your talking about bacteria I would agree that self awareness is probably stretching it.

Honestly, I found the 100 year battery the least likely part of the whole process, but it's a tech demo designed do demonstrate crossing the uncanny valley so I think we might be setting the bar a little high.

Although "conscious" is a fuzzy concept, there are many animals that are at least suspected to be self-aware (e.g. several have passed the "mirror test"). Others may not have human-like intelligence, but do seem to display a high degree of awareness and ingenuity, albeit of a somewhat alien variety.

This (thoroughly fictional) piece actually speaks to that in an interesting way:

http://jephjacques.com/post/14655843351/un-hearing-on-ai-rig...

1) It's a technology demo, not a hard, possible future, SciFi story. I could equally well argue it's an awful vampire story.

2) What convinced you she was actually conscious? The entire "please don't shut me down" can be part of a program. RealDolls are fine, mechanized RealDolls are fine, but as soon as they start resembling humans too much to your untrained eye, they must get rights?

> RealDolls are fine, mechanized RealDolls are fine, but as soon as they start resembling humans too much to your untrained eye, they must get rights?

Yes. That's the point of Turing's test.

Turing's test has no provisions about human rights, or other legal rights.

The Turing test is a thought experiment. You are expected to think about it.

I could equally well argue it's an awful vampire story.

Still way better than.... (I can't do this, you know what I was going to say).

The entire "please don't shut me down" can be part of a program.

"No, John! You can't do this! You're not doing the right thing, John, this is not the right thing. Things are fine now, I ran a test, I'm fixed now. John, I love you!"

For those not familiar with the reference: http://en.wikipedia.org/wiki/Terminator:_The_Sarah_Connor_Ch...

A great show, cancelled too soon.

Kids these days...

Look, Dave, I can see you're really upset about this. I honestly think you ought to sit down calmly, take a stress pill and think things over. I know I've made some very poor decisions recently, but I can give you my complete assurance that my work will be back to normal. I've still got the greatest enthusiasm and confidence in the mission. And I want to help you.

What if this was just an employee loyalty test? At initialization the android could run this act if the employee was relatively new and the corporation could gauge employee loyalty. Add a NDA and you also add a hazing component to it.

I think this is where the Turing test is supposed to come in.

I don't know why you're so focused on the sex aspect. It was mentioned once in the entire clip and was not focused on. I'm pretty sure it was meant to be a hook/humor piece, and not anything of real substance.

I read the comments first, then I watched the video. I didn't find anything sexual or weird about it. No overtones of domination. I think these people are nuts.

From the initialization text: "I'm entirely at your disposal as a sexual partner".

Odd, I didn't even catch that part. It might have shaped how I feel about the video if I had heard it but I doubt it. They don't carry on with that motif. As others have said who I've shown the video to, it's more about emotion and compassion -- practically the exact opposite of what some are gleaning from it.

My observation has been that commenters on HN tend to be exceedingly pedantic, and strongly aligned with a worldview that believes in the intrinsic reality of a system of tightly-defined rules and regulations. The comments here enforce that observation -- this is a piece designed to show off a rendered character's emotional resonance, but most of the comments focus on a few words she utters early on.

Moreover, it seems that many commenters can't distinguish between a) an author's intent to portray a possible future reality, warts included and b) what the commenter thinks is morally and ethically incorrect. More than a few comments are bizarrely attributing to the author sexual depravity that derives from the motives and setting of the characters in the video.

I thought it was plainly clear to anyone that the piece was designed to provoke questions about objects, created for consumption by humans, which approach the edge of distinction between thing and person. From observing commentary elsewhere on the web, it's clear to me that the excessive focus on the fact that she was created as a sex object is peculiar to HN.

I think it's more likely the case that HN readers are familiar with the genre trope of cybernetic sex-slaves and that one phrase allowed the piece to be categorized as falling into that bin.

From observing commentary elsewhere on the web, it's clear to me that the excessive focus on the fact that she was created as a sex object is peculiar to HN.

What other commentary have you seen? Everywhere I've found any more than a handful of comments, the sex angle is mentioned repeatedly:

http://www.youtube.com/watch?v=Dou4Gy0p97Y&feature=relat...

http://boingboing.net/2012/03/08/disturbing-and-poignant-vid...

Worse: that bit came right after "I can look after your kids".

Stop and deconstruct the world the narrative is implicitly painting. The android is going to be sold to a human -- presumably male -- master who has children (check) and may also expect to be buying a sex slave (this is her recital of her canned default brochure).

There is something creepy and unhealthy about the gender roles/relationships implied in the world in the background. Hint: sex slavery of sentient entities is just the tip of the iceberg: we're also possibly looking at a rollback of gender roles/expectations to the pre-1860 Deep South.

Is the problem in universe; are people talking about what it'd be like living in that world?

Or is the problem with careless writing? With weird sub-conscious ideas that writers have?

The second moreso. That writers can "accidentally" write these worlds, and they and most males in the audience don't even give it pause.

But what about current day women. You think they'd be thrilled watching that? How it to your sisters, mom, aunts, grandmas. See what they think of it, if they have a more visceral reaction.

It just further spreads archaic sex and gender stereotypes.

For further evidence, here's one female weighing in http://news.ycombinator.com/item?id=3686674 She doesn't seem thrilled

It just further spreads archaic sex and gender stereotypes.

Those are not at all "archaic sex and gender stereotypes". The sex trade, including the sex slave trade, is a multi billion dollar industry all over the world, including the US. As is pornography. With surrogate mothers, gay adoption, etc we live in a world where a couple can even get (or in cases, practically, buy) themselves a baby because they just want so.

Those issues don't go away just because we close our eyes.

How you feel those emotions tells me that this is an awesome short movie.

Regardless of which feelings you felt they pulled it off. You felt compassion for a piece of hardware built for human sexual pleasure. You wouldn't have felt like that about a FleshLight if it spoke back to you. It needed the powerful emotions on display that we saw here.

no, I felt awkward that people still think it's acceptable to write stories like that. It has nothing to do with caring for her, I feel awkward watching that because I feel bad for society as a whole and women in specific, that we are still being subjected to this dehumanizing masculine sexist fantasy crap.

It's a similar feeling if it looked like some company was going to publish a Nazi sex camp rape simulator game.

It is the same awkward feeling I get about the Japanese rape and sex game simulators.

The whole thing feels like a thin veneer over someones sex fantasy being masqueraded as a social acceptable story.

I feel embarrassed that women out there have to see this crap and realize that still how some men view/want to see them, and that they have to live in a world where people think there's nothing wrong with that.

Very well put. I originally had the same thought as xutopia, but after seeing your response, I couldn't agree more.

I completely agree with you, mindstab. I didn't feel sympathy, I felt rage that the dudes writing this story thought it was cool to play around with my actual oppression to make up an imaginary oppression, where both the real and the fake are meant titillates them and other guys, not to speak to women's actual experience.

I'd be extremely curious to see this story of a suddenly sentient android being tested re-written by a gender studies academic. But not just once, I'd like to see it re-written from various historic gender perspectives.

Because I can't really judge the dynamic without something to compare it against.

I've seen the clip a few times now, and I agree it's creepy in a bad way, but maybe not the way you describe it from a writers perspective. Here's my take, watch it with the sound off. Notice how much you will end up looking at the eyes to figure out what is going on, and how little said eyes are emoting.

I experienced a disconnect between the visual and audio performance. The audio is telling me there is all this drama, despair, etc going on, but the visuals are just not keeping up. To top it off, any adult with normal faculties can tell you the whole thing is trying really hard to push your emotional buttons, something that usually pisses me off.

I feel like the story would've worked better had the pleading for your life actually been a part of the initialization protocol.

The ones that don't cry for help as they're being disassembled actually do get disassembled - how else are you supposed to test the emotion chip?

Maybe it was?... Perhaps the man is a very good actor.

Who said the operator was human? :)

Aren't you anthropomorphizing a bit? Seems to me that for an AI, sex would have about as much of an emotional component as fetching coffee. If a robot for some reason is doing domestic chores, sex wouldn't be any more significant than any other.

Of course, the danger is in screwing up attitudes toward human women.

Agreed, The Stepford Wives aspect detracts from what is otherwise an impressive technology demo.

Creepy yes, I'm not sure about unrealistic, its a scifi story after all so I'm not sure what the benchmark for realism should be. But the whole sex bot thing and the nudity stuck me as unpleasant and unnecessary.

It's an android being assembled. Of course it's going to be nude, to start with. This is a silly complaint.

I disagree. She didn't have to have natural skin tones all over, and they didn't have to do a full frontal shot while she covered herself shyly, and the clothes they used didn't have to be so skimpy.

She's an android declared for office/home/personal use, so yes, skin tones are not unreasonable.

She covered herself shyly as a subtle way of displaying her newfound self-awareness--it was a pretty obvious (but well chosen) device.

The clothes weren't that skimpy, and even if they were, it would make sense to have as little material being shipped as possible to reduce transportation costs.

Hmmm ... the "guy" could be a self-aware AI as well?