Sora might have a 'pervert' problem on its hands | Hacker News Reader

Sora might have a 'pervert' problem on its hands

businessinsider.com

45 pointszdw90 comments

> And how do you stop people from making fetish content of purely AI-generated characters that aren't cameos of real people? Does OpenAI want to stop that? Maybe OpenAI thinks it's fine for people to make belly-flation or foot-fetish videos as long as they're not of a real person.

I can't figure out the tone here. Is the author suggesting we should stop people from creating fetish content of purely AI-generated characters? OpenAI might want to for business reasons, but surely there's nothing inherently wrong with using AI for fetish content. Should we also stop people from drawing fetish content with pencil and paper?

A good-faith reading of the sort of suggestion the author never made is for the blanket opt-in consent to allow other users generate images/videos of you to be segmented into separate consents for PG, adult, fetish etc.; also face/whole body. A very clear consent form that tells them upfront "If you consent to users generating fetish consent of you, here are some examples of what's allowed and forbidden".

> Is the author suggesting we should stop people from creating fetish content of purely AI-generated characters?

Presumably not, but she's farming outrage rather than suggesting any fix. In the above suggested setup, people could then generate fetish consent from the much smaller set of users who consented to have fetish consent generated from them. But then of course they might expect some royalties or revenue-sharing, or at least identification/attribution/watermarking so the depicted user could drive traffic to social-media. OpenAI is skirting around not just segmented consent but any concept of revenue-sharing (i.e. OpenAI wants to dip its toe into OnlyFans territory, but without any revenue-sharing or licensing deal with creators).

> OpenAI is skirting around not just segmented consent but any concept of revenue-sharing (i.e. OpenAI wants to dip its toe into OnlyFans territory, but without any revenue-sharing or licensing deal with creators).

OpenAI is still doing basic experiments with which product offering are well received by users and/or work well and which are not. If some data provided by users (e.g. photos depicting the user) are seen to be very essential to the success of the AI-created content using this data so that OpenAI will likely loose an insane amount of money of these users leave (I think this is rather unlikely, but not impossible), then OpenAI will think about some concept of revenue-sharing, but not before (why should they?).

Pointing out that something feels a little creepy, while explicitly stating that you don’t yuck other people’s yum is hardly “farming outrage.” We collectively need to have earnest conversations about how how emerging technologies affect our experiences, and her tone is pretty middle-of-the-road.

Intentionally opting into an AI platform where we would already foresee a subset of users would start generating sexual content of other users without their knowledge, then professing to be surprised about it, seems contrived. OpenAI had just allowed porn and violence, and we already know a subset of their users will obsessively push the guardrails (plus there's a financial incentive to).

The only parts of the article that seem to be news are a) OpenAI's blanket consent is very broad and doesn't warn users what might be done with cameos, or segment consent into various different types of content use (as even 25yo modeling sites do) b) that subset of users will bypass the guardrails and c) OpenAI doesn't close the feedback loop by notifying the users in a) what the users in set b) are doing, let alone allow revising or revoking consent.

But why is the conversation only about b) (the predictable bad behavior by users) rather than a) and c) (feasible solutions)?

Notopoulos correctly remarks: "part of an overall pattern of how OpenAI has approached the concept of copyright and intellectual property: asking forgiveness, not permission."

By the way previous (non-sexual-content) incarnations of this sort of issue are 2019 when Clearview scraped 3 billion images non-consensually from people's social-media and state DMVs [https://news.ycombinator.com/item?id=35421117].

Or the previous 2024 OpenAI/Scarlett Johansson-sounding voice shenanigans. Or the existing proliferation of AI porn elsewhere.

Bow does other people doing things affect her, though?

> but surely there's nothing inherently wrong with using AI for fetish content.

It's taking work from Onlyfans, Artists who draw fetish content on commission (usually by copying other people art style), fanfic writers (who copy writing style, characters and setting of other people) and other organic and free range fetish content producers.

I see that position, but also, almost everyone (99% of people?) generating fetish content with AI would not otherwise be paying artists to commission that content.

Many girls believe that there is both inherent harm and external harm in the creation and consumption of sexual content. Arguing that AI content is inherently harmful is tricky, but can be replaced with yet more external harms.

Many religious extremists, gender unrelated.

I don't think she's passing any judgment here - she pointed out earlier in the article that she knows people are into weird stuff, and didn't want to "yuck their yum."

I don't think one gets to call these people 'pervert' in the title, then claim that you don't 'yuck their yum'. They yucked their yum before the article even started.

Titles frequently aren't in the control of the article writer. Don't know if that's the case here or not.

The problem is adults contribute to turning public platforms lewd. So one lewd person on Instagram leads to many, leading to a lewd platform. This becomes problematic when the children uptake it. It’s not really too different than prostitution appearing near general-purpose places, it turns it into a red light district.

A lot of social media is a sex platform, and it got mixed up in this way because there’s no talking adults out of being lewd in public.

Present-day adults have been raised in a society where overt sexualization has been the norm for decades. I see normalization of lewdness as a logical consequence of 60 years of sexualization in media, and the growing resistance against it as the normal swinging of the pendulum.

I wish we could talk adults out of it, but many decades on this earth have convinced me that’s just not going to happen.

It's not easy or sure, but behaviors do change, dramatically sometimes. Youngish people have significantly less sex than before. People smoke a lot less in some countries, as one simple example.

Social media is still immature. We'll develop norms around what is appropriate.

From the article

> How do you stop people from making fetish content of purely AI-generated characters that aren't cameos of real people? Does OpenAI want to stop that?

I'd say a resounding no. Didn't Sam Altman announce a little while back that they're exploring allowing ChatGPT to be used for erotica / NSFW generation?

Who could possibly have seen this coming.

Why are teams always, always so naive?

This is kind of a side issue but I think the time has come where videoing in public and using images of people without explicit consent should be outlawed. It used to be okay when the only medium was TV or movies and it was hard to distribute but now I think it has become a nuisance at best and dangerous at worst. The only thing I would make an exception for is actual journalists with some sort of credentials. But posting videos on social media of unwitting unconsenting people should be outlawed in the age where people with phones can upload videos in second.

In a world where absolutely everyone has a camera in their pocket I can’t see how you’d ever be able to enforce this.

Terrible argument, you should say because of generative AI where anybody can fabricate any image/video/audio of anyone in any context without their consent we urgently need new regulation of social media platforms to accommodate this profound change in our reality. Then make your argument.

You have no expectation of privacy in public spaces since forever, that is not a problem. Because nobody can photograph you stabbing someone and uploading it on social media without you actually stabbing someone. This is now different, because anyone now can make that photograph of you stabbing someone and post it.

That must be your argument.

And it must be on the social media side, because in X months some open model on GitHub is gonna make every watermark or cloud-based safety feature meaningless anyway.

Making fetish porn is the no1 use case for AI since inception.

Just force Sora to avoid photos of real people. Instead, synthesize a "generic" face. This can be done with training, create a database of photos to avoid and train Sora on that.

Is it actually easy to train the AI on all the faces so it can make pictures that look like humans while also training it to not make pictures that look like any specific human?

the whole point of sora is you can generate photos of yourself and friends...

The app allows you to control if other people can generate photos of you. If the author doesn't want other people to make these photos, disable public video generation...

Wait a sec, you have to specifically opt-out of this? OpenAI needs to be bankrupted for their massive copyright infringement.

Sigh, it's in TFA.

> I've allowed anyone to make "cameos" using my face. (You don't have to do this: You can choose settings that make your likeness private, or open to just your friends — but I figured, why not? And left my likeness open to everyone, just like Sam Altman.)

no she opted in

> I've allowed anyone to make "cameos" using my face. (You don't have to do this …)

… and this is probably where the article should’ve ended. Or in fact, where the author should’ve realized there didn’t need to be an article at all.

People are weird and gross. They do weird things that we would often prefer they not. Sora provided a tool to avoid that weirdness. Use it.

People always have been pervy and gross in private and society has always provided outlets for them to do so while preserving their privacy and dignity.

However, using a wannabe AI social media platform to engage with this stuff (and said platform encouraging you to do so), is crossing several uncomfortable lines for me.

It crosses lines for me too. So I won’t be allowing people to use my face. QED.

I mean any sort of involvement in this in any way, either making stuff or being used as a face model or anything. This is semi-public and content you make is associated with you, and so is the content made about you.

This does appear to be her shtick. Engaging in a thing so she can report upon it.

https://www.businessinsider.com/threads-meta-engagement-rage...

That is something of a valuable service. People might see that option and think “well, why not? What’s the harm?” This way they can learn what the harm is, without having to be the person being harmed.

I agree that this can be a method that can provide insight. It's also one of the areas where journalistic ethics are most strongly considered.

A journalist has done good work if they report on their ability to smuggle a replica bomb onto a plane. It's a bit hazier if they smuggle a real bomb on board because it puts people at risk. They shouldn't blow up a plane to show how easy it is.

I didn't offer any judgement in my comment as to the ethics of this particular reporter, just noting that was the style that they do.

The claims she purported to represent Meta in public statements would, if true, count as unethical journalism. I don't know of the accuracy of those claims, so at this stage I would remain undecided but wary.

That’s called Journalism

All evidence points to the author trying to make this a moral panic, especially with the emphasis on "real women" being used to generate these (despite the feature meaning they have consented to it)

Will you be a little outraged though when they use real women without their consent?

If we know anything about software in general (and AI specifically) getting around roadblocks is often a fairly simple thing.

Yes. And we also have documented cases where generative AI, in the hands of people with serious psychological issues, is significantly accelerating those people's loss of control of those issues, to very negative outcomes.

The fact that the AI industry is apparently littered with incredibly immature guys who perceive themselves to be Randian superheros does not reassure me that this tool is going to be better.

What additional solution would you propose?

I propose a subtractive solution: that we ban this category of congenerative video products entirely, or consider criminalising any use that reproduces the appearance of a known person without their explicit contractual consent.

(I really don't care that this suggestion typically provokes what amounts to libertarian screeching)

"congenerative" was a weird lack of sleep brain thing, incidentally, but I think I like it. It has many useful allusions.

> Will you be a little outraged though when they use real women without their consent?

It's a complicated issue that I've considered many times before. If we deem deepfake pornography unethical because it creates images/videos that look like real people, what does this mean for "lookalike" pornography, featuring actors done up (or who just naturally look) like famous people?

For example: Let's say Person A has a friend, Person B, who looks like Person C. Person B consents to Person A using artificial intelligence to generate pornographic images of them, which in turn look like pornographic images of Person C. Should Person A need consent from Person C?

Is it really that complicated? I'm not sure the slippery slope towards a potential future with too-generic porn faces is really all that convincing against the very real and ongoing mass distribution of a nascent, poorly regulated (or even understood) medium that features torrential quantities of--let's not mince words--sexual exploitation.

> poorly regulated (or even understood) medium that features torrential quantities of--let's not mince words--sexual exploitation.

Surely you'd be in support of the use of artificial intelligence in porn as no real people engage in sexual exploitation?

Engaging with sexual exploitation includes its consumption, and it absolutely includes its creation. I don't have any problems with content, generated or otherwise, that includes informed consent from all models involved. But that's not what's being described in this article. They are specifically talking about photorealistic imagery generated using the likeness of individuals who did not knowingly consent to such depictions.

>Will you be a little outraged though when they use real women without their consent?

Probably not. AI slop doesn't really "go viral" except when it is super ridiculous like shrimp Jesus. Most people generating AI slop porn are likely in the 10s of people who will see it. If someone generates porn with my face on it and I never even know, how does this harm me? Why should I care?

This seems like you are not fully thinking this through, either intentionally or not. Benefit of the doubt, I will add to your question about why you should care with more questions: what if it's someone you know? what if they want to then sell the generated content? what if it was your political enemies? what if it was your boss? what if it was someone who was stalking you or made you feel unsafe?

>This seems like you are not fully thinking this through, either intentionally or not. Benefit of the doubt

I am not, that is why I'm leaving it to replies to convince me otherwise.

>what if it's someone you know?

Why not start with what if it's me? If I don't know about it, implicitly, I don't care. But if I do and it bothers me, AI companies have already been given free reign over copyright law so I don't really have any recourse here. Can I sue Sam Altman? No, because if that was possible, someone like Studio Ghibli would have done it and made a billion dollars by now.

>what if they want to then sell the generated content?

If it's generated by AI, then the courts have already stated it can't be copyrighted, and is therefore unsellable. The first person you sell it to can redistribute it for free and there's no way you can stop them.

>what if it was your political enemies?

I'm not political, which upsets a lot of people. Just tell people "I'm not voting, it doesn't matter" and watch them lose their minds.

>what if it was your boss?

I'm not really sure why I should be fighting my boss's battles.

>what if it was someone who was stalking you or made you feel unsafe?

People can stalk me without AI. I'm not sure how AI changes this.

Have you been in a high school in the past two years? "Your classmates will generate AI porn of you and share it amongst your other classmates" is rampant.

There are dozens of scenarios where this will completely ruin your life. One example is someone generating prom of you with underage children and sending it to everyone you know -> life ruined. You can already easily do this with open source models

What's stopping a really fucked up person from doing this with Photoshop already?

It’s a lot more to do it with photoshop.

> Most people generating AI slop porn are likely in the 10s of people who will see it. If someone generates porn with my face on it and I never even know, how does this harm me? Why should I care?

You would presumably care if one of those "10s of people" was a family member or peer.

Maybe not caring is enlightened of you, but it shouldn't stretch your imagination to consider why others would.

>You would presumably care if one of those "10s of people" was a family member or peer.

This is an interesting angle. Maybe someone might try to blackmail me with AI porn of me cheating on my spouse? This could also be a great usecase for divorce attorneys. But it would be easy enough to do that without AI. There's the old joke:

>Once a year I send out 1000 Valentine Day cards signed "Guess Who? XOXO".

>Its a cheap and easy marketing plan for my divorce attorney law group.

Isn't generating fetish content a legitimate business model? Doesn't anyone remember the song,

>The internet is for porn

I actually think this is what's going to happen with AI once the easy money dries up. They'll quickly race to the bottom selling porn generators. AI slop porn already seems like the majority usage after homework generation.

The article is describing a feedback loop: very few women consent to have their face be usable -> the number of perverts vastly outnumber the number of women so each woman, who has posted nothing nsfw, or even suggestive, gets dedicated attention from many perverts -> other women aren’t comfortable allowing their faces to be used by large numbers of perverts-> very few women consent to have their faces be usable.

This is all independent of what is and isn’t a legitimate business model, it’s a social dynamic. It’s also a pretty familiar one: it shows up everywhere from nightclub bouncer policies to the dynamics of early 2000s irc rooms

The number of perverts vastly outweigh the number of women? Um?

You can also use photoshop to glue a woman's head on a porn star's body. The results are about the same as AI slop. AI is just faster.

Don't even bother to look at r/grok. When you have a waifu anime avatar… yeah, it went exactly where you would expect it to. Apparently XAi is reigning in the NSFW and people on the sub are livid.

Well thanks for that rabbit hole... 20 mins I'll never get back :P funny tho

might? ascii art has a pervert problem.

Yup. Likewise cinematography, photography, fiction writing, spoken language, and cave paintings, all have had a pervert problem since day one.

humans are perverts. who knew?

I have no idea how I would feel if someone used my face in fetish or sexual content. Never happened. It’d probably make me uncomfortable. But, I imagine I would be ok if I grew up in a culture where sex wasn’t as much of a taboo. Maybe I would find it flattering.

My mental test for deciding whether something should be illegal or unacceptable is questioning if any one would see it the same if religion never existed.

During #metoo I remember reading an article where the author was uncomfortable with the “drug fueled sex parties in Silicon Valley.” They basically didn’t want consenting adults to do drugs or engage in group sex. The argument against fetish content with AI generated characters reminded me of the #metoo author’s discomfort with the drug/sex freedom of the Bay Area. The article about Sora sounds like the author is uncomfortable with people generating fetish content, regardless of the content featuring real people or not.

It’s sad that the liberals now include the prudes/conservatives.

I don’t think one is a prude or a conservative for not wanting AI generated porn of themselves to exist.

I also don’t think one is a prude or a conservative for thinking there are consent and power issues around anything that commingles sex and the workplace.

Things can be unacceptable without being illegal. Things can even be unacceptable without needing to be banned or privately controlled.

My bar for what should be unacceptable is a lot lower than my bar for what should be illegal or privately banned.

Making weird pregnancy fetish videos of real people without their permission is definitely unacceptable. I have no issue with the idea that anyone doing that should be shamed.

I was pretty clear about my agreement with the real people AI fakes issue. And pretty clear on what I considered to be prudish. Would mind reading my comment and editing your response? Or you’re responding to someone else?

Edit: the fact that the author is right to be uncomfortable with their face on generated fetish content, doesn’t make their stance on fetish content with generated characters less prudish. They can be right about one thing, and prudish about the other.

Journalism might have a "bullshit content" problem on its hands.

Yep, guilty as charged

So the author specifically allowed their face to be used for content and then is surprised people acted on it? This is silly imo.

People should just never even allow their face to be used for this.

This reminds me of the "Joanne is awful" episode from black mirror. Exact same story.

You’re treating this article as an attack when it’s not one. She turned it on to see what would happen, because she’s a technology reporter, and then reported on what happened. And her reporting isn’t even all that negative.

Tbh if you are sufficiently famous or have a bunch of pictures of you available online you don’t need to opt-in.

So this is silly yes but people can still easily make deepfake porn of you from any photo available with other oss tools

Sure, but then they’re not immediately public on Sora xD

Rule 34

But the idea that someone was making a video that had some potential sexual gratification element made me feel fairly icked out.

Clearly the author has never visited 4chan... but I think seeing others make such content with your appearance should be taken as flattery.

More seriously, I hope this flood of easy video generation will cause people to more easily realise how they can be persuaded, and increase skepticism of evidence in general.

I think seeing others make such content with your appearance should be taken as flattery.

Post body or gtfo

(for those lacking context, this is a callback to a 4chan trope that is inextricable from OP's argument)

> but I think seeing others make such content with your appearance should be taken as flattery.

I think maybe you're an adult male.

Having actually talked to some female friends about this, I'm pretty sure that women in general don't take so well to the idea of tools that might be used to encourage the fantasies of the men that already have a dangerous interest in harming them sexually.

Whether the Valley thinks that's their problem to solve, I doubt. But making a joke out of it is pretty fucked up, dude.

ETA: even women who have done some modelling and are a bit more aware of the way those images are used are at least somewhat concerned about content that can make them act and speak like puppets. This is at least as much about consent as it is about content.

ETA2: I am rate-limited for being an argumentative sod in the past so I will finally edit this to note that 1) I am replying only to the sentence I quoted which has very troubling connotations, and 2) I really think a lot of people here seem not to have read Julian Dibbell's crucial 1993 article "A Rape In Cyberspace" and it really shows.

The OP never at any point suggested "harming them sexually". Her objection was to people making fetish content of her (or other real people) without their consent, which is entirely a reasonable objection. Let us not misrepresent her.

(As commented elsewhere, the author expressly opted-in, apparently with the intent of generating ragebait to write an article about.)

I know some female friends who wouldn't care. Because they actually understand the difference between fantasy and reality.

> tools that might be used to encourage the fantasies of the men that already have a dangerous interest in harming them sexually.

This assumption sounds like being taken from some feminist manifesto.

OP did use the word “might”. Not saying every man has that dangerous interest but are there men out there with sexually dangerous intent? Without a doubt.

More to the point, there are men out there with unhealthy interests that haven't and might not accelerate to something worse.

Much like when that AI executive dude started talking about layers of hidden reality or whatever it was, that some LLM helped him "find": people were clear that whatever his problems, he might not have blurted that shit out loud, or even developed those thoughts as much, if it were not for the reassurance loop of whatever tool was helping him go a bit more mad.

We understand what happened in his case, right? Perhaps he was keeping that under control and then wasn't, because it was all so plausible.

Now imagine it being video of some young woman realistically depicted doing things she has not consented to do, in the hands of a man who is obsessed and is just keeping that under control. An obsessive fan, for example.

You can just as easily imagine that going the other way?

I get your point, but I’ve never seen any research into whether this material makes people more or less likely to actually perpetrate crimes related to it.

A chat loop is a bit different from a static video/photo.

Sure, but by that reasoning there’s women out there with sexually dangerous intent too.

Not at a first approximation.

One in five women in the US experience rape or attempted rape during their lifetime, overwhelmingly by someone they alreay know at the time (https://www.nsvrc.org/statistics).

And the majority have experienced some stalking in real life.

It's a totally reasonable, evidence-backed, dare I say it plainly obvious, non-feminist assumption that it might be used that way, and you know it.

Reply on news.ycombinator.com