Korean women remove pictures, videos from social media amid deepfake porn crisis
channelnewsasia.com
channelnewsasia.com
Curious to know if 20-30 years ago, as graphic design tools got better, were people freaking out about being able to superimpose another person's face on a the body of p*rnstar?
Superimposing people's faces on to porn star's bodies happened/s, but does anyone care really even notice or care about it now days? For the most part, of course not.
So why should we care about porn videos? What makes them so different? The whole concern smells like typical media hype (i.e. yay - we get to write about AI + Porn - instant clickbait) that nobody will be able to stop and nobody will even care about in 1-2 years: e.g. "oh, yah, another deepfake of <(politician|actress|colleague^)> in a porn film - how incredibly creative".
^ Incidentally, this was made light of in the British version of The Office where a modified image of the boss, David Brent, was sent around the office. If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.
> If it's joked about harmlessly on mainstream TV, it's probably not a huge concern.
There are plenty of mainstream TV jokes about horrendous things. Our ability to joke about something is a poor way to gauge how concerned we should be.
Is your concern that recipients would think it's real? Or is it that people will get an email/chat message containing NSFW material? (because the latter happens now, and is nothing more than background noise at most). What is your actual, specific concern? (genuinely, I have no idea, but want to try to understand - to me it feels like panic about a problem that isn't actually a real problem).
But I think there's a difference between a crud photoshop picture that a person has on their PC, and a (more or less) convincing deepfake video of a person being broadcast.
I think you're likely right that there is an element of panic. I don't know a lot about Korean culture, but I imagine the concern is that deepfakes can have real-life negative impact on people's lives, either because people think it's real or because it creates a hyper-sexualised social environment that endangers women. Both are worthy of concern in my view.
There's still no case for why fake video is apparently so much worse than fake images. Both achieve the same effects when sent maliciously to a relative or contacts list.
I'll (try to) steelman the case for why video's worse: because people don't realise how easy fake videos are to make. But in 1-2 years... meh... most of society will have it figured out.
Oh but the average guy isn't that perverted? Yes they are. We know this from the vast amount of porn they consume and of what type they consume.
What perverted things someone does with photos in the privacy of their own home seems like it should be their own business to me.
I think the bar is roughly the same, all things considered. If you're going to photoshop, then it's largely a matter of combining two images with a sufficient amount of masking and blurring to make it appear good enough.
With image generators, you've got a couple different options. The more trivial would be doing img2img on your target, the more complex being making a LORA.
Knowing how tech savvy the average Andy is not, I would say anyone capable of going through the steps to do the AI method could, with some brute-forcing, also manage the Photoshop method. Setting up a local instance of any image generator is beyond the majority of the population, so the closest most can get is paid online GUIs that offer an existing service that they first must become moderately proficient in for good results.
Only real difference is that anyone who can do it is much more capable of scale over time than previously. With enough Comfy nodes, you could probably even automate it given face detection and arbitrary images.
Doing that on an image generator is trivial, there are whole websites with the sole purpose of inputting a seed image to be transformed through a prompt, that's a much lower barrier of entry for someone trying to make a fake nude. And it scales, one can do this in a batch in a few minutes/hours, doing that manually on a photo editor requires a lot more effort and knowledge.
I don’t think the bar is in the same state.
IMO it is only a matter of time before this same issue comes to the forefront in the US (I'm kinda surprised it hasn't already become a bigger topic). I could easily see custom celeb LORAs/emmbeddings and realistic SD/SDXL checkpoints being banned outright for this reason. Yes, it would be an infringement on free speech, but weighed against the interests & safety of the public, the courts might see a move like that as reasonable.
It's that or we will come to accept some level of societal degradation due to the abuse/misuse of AI tools becoming normalized.
I do think it’s a hard problem (especially if we want robustness to compression etc) but amenable to research.
It has already been inching that way for a long time, but generative AI definitely is adding new fuel to the fire.
It's just the new reality. Porn is not even the worst thing. If anything it can make real photos/videos harder to get viral.
But augmentation will be (if not already is) used to make very tiny adjustments, like you make one candidate/celebrity face slightly more symmetrical and the other one less, change their gait just a little bit. These changes are non-perceptible, are not interesting enough to make a scandal, and they can have huge impact on how people perceive other people. I mean, I am not suggesting that it isn't a puppet show without those, but the game is evolving.
Many do not know it’s happening. Those that do and aren’t chronically online find it horrifying and even traumatizing.
You can tell how few women are involved at the key stages of building new tech because of how rarely obvious harassment vectors are accounted for during development.
The problem I have is with trying to get men to care about it because it affects their daughter or whatever.
Interesting point to think about- it certainly would have helped. But I also genuinely don't see how someone developing this kind of technology, who has had even a brief glimpse of online culture any point in the last 20 years, didn't immediately see what it would be used for. It almost feels like "progress" with malicious intent. Or an extreme case of apathy towards societal consequences.
>The problem I have is with trying to get men to care about it because it affects their daughter or whatever.
Lot of people are too self centered to care about any issue unless it directly impacts them. If it gets people thinking about how it affects their loved ones I don't think that's necessarily a bad thing.
It's just shitty both to men and women. Men don't care about an issue unless it affects their daughters? They don't have other women they care about? Mothers, friends, mentors? Or for that matter sons who are also vulnerable to abuse and manipulation? The men I know are better than this.
And it positions women as being just a motivation for the actions of men, whose right relationship is to be protected rather than being actors in their own right. If you find this tactic sound then feel free to use it but I think there are better ways.
But it's an issue that has both then and now predominantly affected mostly women. Sadly I reckon many men will not care about it, or even be aware of the issue, until it impacts a woman they know.
From what I can see the issue here is that this problem with AI can potentially become much much more widespread because the amount of work required is pretty much none and gets lower as models get better. It sounds like someone using deepfakes to ie extort people can just cast a much wider net by just scraping/dumping masses of images into a generator to create a multitude of convincing pictures of many different people instead of having to focus their time and effort on picking a single target and creating fakes manually.
We didn't solve teens doctoring inappropriate images of their classmates by banning Photoshop or accepting it as the new normal. We solved it by passing laws which criminalise the behaviour and enforcing them to deter the behaviour.
Technology influences society. But plenty of social mores are almost universally respected despite being difficult to enforce. Possible and difficult to enforce doesn't force permissibility.
I don't accept the idea that "passing laws and enforcing them to deter behavior" was the cause of the lack of issues, because to be a deterrent us teens would have had to been aware of the laws.
Same. And same.
Drawing inappropriate pictures of your classmates was creepy before. Making Photoshop or AI porn remains so now. Most people won't do it. But there is active social reinforcement that keeps it from becoming "the new reality."
> don't accept the idea that "passing laws and enforcing them to deter behavior" was the cause of the lack of issue
Fair enough.
The jargon was cyberbullying, and there was absolutely legislative activity around horrific examples. But the principal mechanism of action probably wasn't a direct fear of getting caught and punished, but the prompted discussion reinforcing and extending the norm.
There will be a time when the Trump "grab them by the you know what" style scandal will just be met with sceptism.
Where do you go when even video evidence can't be seen as the truth?
If the video is edited or comes from a source that doesn’t have trusted keys then it’s no better than some random tweet
I think the reality is that photographs and video are now like text, you trust the source or not.
I can write about Cleopatra's ability to snowboard and play Xbox, or I can make a photo with the caption "Cleopatra doing a 1080", and now I can make a video of it (probably).
I can also give you a certificate/key/blockchain whatever to prove it's "valid".
I thought we have general laws around sexual harassment and revenge porn which criminalize the point of harm.
I bring this up because most of the enforcement difficulty seems to stem from a desire to to control behavior which would otherwise be undetectable.
I believe we already have means of enforcement for students sharing pornography of their classmates.
At least as far as criminal harassment goes. Schools can adopt more restrictive policies but the consequences would only be suspension, expulsion, etc. not criminal charges.
What if they do so publicly?
If the police actually tried to prosecute them for harassment, the question the courts would be asking is, "could Jane reasonably avoid having to hear it?" Harassment is about being able to not be the recipient of communications you don't want to receive. Calling Jane fat on message boards, twitter, in a bar, etc. would all be fair game.
There's more restrictions if Jane is a captive audience. The courts have ruled that certain settings where people can't easily just leave - namely work and school - have greater restrictions. Again, the question the court would be asking is "can Jane reasonably avoid this speech" and it's a much bigger barrier to leave one's workplace than walk out of the bar.
This is like bringing your howitzer to a gun debate because they are kinda similar.
For the time being, the easiest-to-use tools are centrally run. That makes monitoring and enforcement easier.
Freedom of speech is mainly so that we can criticize the government. How does deepfake porn or impersonation have anything to do with speech?
Freedom of speech is a bigger concept than the First Amendment. Technically curtailing one's ability to make AI porn of your colleagues is a restriction on speech. That said, so is every spam filter, fraud statute and retaliation against a declaration of war. A right doesn't have to be absolute to be good (or vice versa).
Information intentionally communicated by a person to another is speech by the strictest classical definition. (I guess technically the creation of the AI porn isn't speech in the way a falling tree in a deserted forest doesn't make a perceived sound. But it's a schoolyard riddle for a reason.)
Your other examples, yes. This one, not at all.
Freedom of speech is against government intervention, not against private agreements.
Rather in fact an absolute form of freedom of speech would protect against the government from stopping you from forwarding all your correspondence to someone else, and would protect against the government stopping that someone else from removing the spam and sending the rest back to you. Non-government run spam filters aren't just not offending freedom of speech, they are protected by it.
You're conflating the First Amendment and freedom of speech [1]. If you're in my house and I prohibit you from expressing certain views, that is fine by the First Amendment but not in the spirit of free speech.
And as I argued above, since every step in having someone filter your communications for spam is simply speech, it protects spam filters.
You restricted it to "government intervention, not...private agreements." The freedom of speech extends to the private sphere.
> You're conflating freedom of speech with a duty to listen
These are inextricably linked in the classical form. In the Athenian tradition, there was a duty to listen [1]. (Though even then practicality made it far from absolute.) That traces through to the Protestation of 1621 [2] and the Founders' debates around the First Amendment (namely, between Madison and Hamilton).
Practically, of course, no duty is absolute. But a society where everyone screams into a void isn't adhering to the principles of free speech and expression. (I'll note that alongside the duty to listen is an implied duty on the part of the speaker to speak truly [3]. Spam is the epitome of the cessation of the latter revoking the former.)
[1] https://www.stoa.org/demos/article_democracy_overview@page=a...
I'm not sure where you find a duty to listen in Athens, my best guess is you are pointing to the duty to attend the assembly, which fits this form of "parliamentary privilege" rather than a broader duty to listen. To contrast, by my understanding political commentary in the form of plays and the like were quite common in Athenian culture, and I'm fairly sure there was absolutely no duty to pay attention to those. I'd also note that even with the assembly "the crowd might raise a clamor and refuse to listen to a speaker advocate an unpopular proposal" (your source) which suggests little duty to actively listen even there.
The protestation of 1621 is directly related to parliamentary privilege (and why I labelled this the way I did, though I don't think the label fits perfectly, since it applies to many smaller government meetings than parliament).
I can't say I'm familiar with the first amendment debate you are discussing.
> You restricted it to "government intervention, not...private agreements."
I don't think this is the cause of our disagreement, but I'll grant you that that phrase was slightly inartful. E.g. there are authorities other than governments, like a local gang, who could impinge on freedom of speech with threats of retaliation, and to the extent that private agreements are enforced by an authority it's possible for them to do so.
Technically, parliamentary privilege derives from the Athenian model of freedom of expression. In the case of the latter, the privilege (and duty) extended to the whole demos. Politics were never “off.” (Well, unless you were a child. Or a woman. Or a slave. Or a foreigner.)
One can trace it from there to the Roman Republic, and most pointedly, from folks in the early Empire lamenting how private society proactively narrowed the Overton window to align with perceptions of Imperial preference. (The Senate floor was muzzled by dissuading private criticism, officially and informally.)
Those “classics” then went to England and France, and then to America and Europe’s colonies, and then to the UN and EU. TL; DR Freedom of expression in Western thought as I understand it.
> which suggests little duty to actively listen even there
It was generally notable enough to be recorded, which suggests a general restraint. Not an absolute duty. But more than a little.
> don't think this is the cause of our disagreement
I don’t actually think we disagree :). We just know more about different aspects of the same thing.
I don't think they're a concern at all, other than that they will ruin the evidentiary value of photographs. They'll do that whether or not the general public has access to them.
The concern about fake pictures of people naked is stupid. The same process mentioned above happens to naked pictures of people: they lose evidentiary value. General access to the ability to make deepfakes not only makes it so people won't believe what they see now, but also won't believe actual revenge porn. It simply ceases to be a problem. You can claim it's a fake, and no one can disprove it. This is a reason for celebration.
The standard response for that has been for people to simply claim that it won't matter, and that people will destroy the life of some young innocent girl if a mean boy spreads fake pictures of her naked. The answer is that people who harass a young girl are guilty of harassment, and should be jailed. You shouldn't be harassing people even if naked pictures of them are real, which they often are. And if you'd attack a person over pictures that you could fake up yourself in five minutes, you're either a) a moron or b) somebody using the photos as an excuse, and it would be a benefit for society for you to be locked away from it.
I suspect a lot of people really worried about deepfake porn deep down are worried about their ability to properly identify and harass the harlots if people won't believe the pictures that you send to their family and friends anymore. From my perspective, they're literally fighting to preserve revenge porn.
This is going to be a monumental issue when the fact that we can't trust any photographs fully takes hold. Maybe watermarking will help with that. More likely people will develop tools to circumvent watermarking technology (either on the imprinting side, or on the verification side).
> people won't believe what they see now, but also won't believe actual revenge porn. It simply ceases to be a problem.
This may be a comforting way to think about it, but it won't work in all cases. Consider instead of revenge porn, the case of CSAM – that will never be a solved problem until 100% of it is wiped from the face of the earth and it is never produced again. Making it less believable because there is so much AI-generated equivalents out there is a not a solution to the problem, it exacerbates the problem.
Full faith in photography was never a thing. If anything, the advent of photography came alongside yellow journalism. Trust is an institution. You can't build institution on solely technology.
Even with CGI being what it has been since the 70s, it's still appropriate to apply Occam's Razor in many cases, and to simply say that because it looks real, it probably is.
I think for a lot of people it's easy to die on the hill of technological progress and freedom at any cost when the consequences haven't happened to you or a loved one.
This is too narrow: it presumes rational contexts where evidence will be evaluated critically, and it assumes a mild, uncreative attacker but harassers are good at exploiting social norms. The damage caused by this kind of harassment isn’t seeing yourself nude as much as wondering or knowing what other people will think, and there’s no way to undo that. Even if people _say_ they believe that the pictures are fake, the victim may be wondering whether that’s really true for the rest of their life.
The other thing to think about is that we’re really just getting started. Most of the deepfakes which the average person has seen are fairly mainstream porn nudes: there’s a huge control issue but in general there are worse things than people thinking you have a star’s body. We’re going to see those worse things soon: it won’t be a porn fantasy but things designed to degrade and more targeted - if the target had body issues, exaggerating them; punishing the person who rejected someone by faking them looking not like a porn star but less attractive and having degrading sex with someone unpopular, knowing that a bunch of people are going to think “well, he would say that”; make some teacher’s life hell by faking them having sex with a student; etc.
Because the attacks are being directed by humans, they’re going to adjust and find what works. Anything which forces an official investigation is going to be especially bad because we can’t just ignore things like child abuse so it’s going to be like SWATing where there’s an ugly period where the wrong teenager can get an incredibly high-leverage response with a single message.
The harm comes from falsely accusing someone of rape, not the deepfake itself. This can be done with or without a deepfake.
As a counterpoint: if a group of boys find their teacher is really hot and they use Stable Diffusing inpaint her clothes to create a fictional naked photo, where's the harm? They could also just use their imaginations to fantasize about what she'd look naked. I don't really see that much different - they're essentially delegating the imagination to the computer.
The typical response is that if the teacher discovers these photos, that discovery could be disturbing for her. I do agree. But at the same time, if these boys walk up to her and tell her "I just jacked off to fantasies of you last night", that's also probably disturbing too. Like the first scenario, the deepfake is tangential to the actual harm: the harm stems from telling another person that you've fantasized about them sexually - not the exact nature of whether that fantasy was produced by a human brain or a silicon one.
As long as it looks convincing, when deepfakes start circulating like this, the object of the deepfake will face consequences far in excess of what you are suggesting. Hell, some parts of the country even obvious deepfakes may get a woman in trouble. Any woman, not just a teacher.
You explain to me how to fix that. There's a whole culture around treating women (and their bodies) as a corruptive moral threat, which has proven completely intractable. God forbid a real photo leak somehow - the woman still gets fired. That shouldn't happen, but our culture seems to insist on it.
> Again, the harm comes from dishonesty not the deepfake itself.
This is a weird argument. "The deepfake is a thing; it has no agency and thus is not capable of any harm of its own accord" seems to be akin to the argument of "Guns don't kill people; people kill people". It similarly misses the trees for the forest.
The person I was responding to said they were not “a concern at all”, but clearly that can’t be true if they make harassment and false accusations easier and more damaging. Similarly, we shouldn’t dismiss the impact of making those attacks stronger - it’s like saying that giving a kid a gun is no big deal because they used to have a slingshot.
You missed the reason why I mentioned that scenario. It’s not just that the targeted teacher gets harassed — and, let’s be honest, that’d be a lot of aggression just below the threshold of serious disciplinary measures – but that something like that would if found trigger a mandatory law enforcement response. School employees tend to be mandatory reporters so they don’t have the option of saying “that’s a fake, he’s probably lying”.
The real harm is that unless it’s a very obvious fake, the victim is going to have to go through a police investigation into their personal life. Want to bet that’d especially be used to target people who have reason not to trust the police because they’re gay, brown, not an evangelical Christian, etc.?
Photoshop and other advances long since killed that certainty, and now we have a healthy distrust of pictures we're not sure where they came from.
I expect videos will now walk a similar path, and life will go on with no major disasters.
Kindly, if you grew up after this picture was taken, and you believed the guideline “a photo never lies,” you were wrong.
https://en.wikipedia.org/wiki/Censorship_of_images_in_the_So...
It was a rule you thought you could rely on.
I'm of course aware that there were forged photos back in the day. But they were rare and hard to make. They were usually paper only. Producing a fake negative was really hard, and a forensic expert could often spot it.
Respectfully, we only wish this were the case. There are, right now, AI generated images and videos of a presidential candidate in a disaster area. People want to believe they are real. And so to millions of people, they are real.
People do the same with faked nudes. They will believe in those nudes because they believe that women are inherently sinful (huzzah for "original sin") and absolutely would do such things.
Oh, and Photoshop never had a dedicated app (or 100 dedicated apps) for creating nudes from regular photos and/or videos. It took a skill a scant fraction of a percent of people had.
South Korea’s national assembly on Thursday passed a Bill to punish people possessing, purchasing, saving or viewing deepfake sexual materials and other fabricated videos.
(Publication of the contents of the files is another matter. I appreciate how this can be cause for prosecution.)
EDIT: I can foresee a dark future where surveillance video is monetized for such salacious purpose and it becomes inescapable as the cameras are everywhere.
* I suspect that, like prohibition or DMCA, criminalizing behavior will lead to worse outcomes than accepting deepfake porn sometimes happens, at least if laws are passed targeting this in isolation.
* I suspect that before acceptance, many people will be harmed, on both sides. Humiliated teen girls. Immature hormone-driven teen boys.
* I suspect that we'll see a range of bigger problems such as real scams where, for example, a deepfake child calls their elderly parents and empties their bank account. Or, perhaps, a deepfake general calls someone in a nuclear bunker.
* Deepfakes are at the tip of technological problems which need solving. We have other things, like the cheap and easy availability of technology to make the next Covid-19. The point is we have the technology to make it in a lab, and the price point is going down by an almost Moore's Law like curve.
I don't see a good solution short of completely rethinking democracy and capitalism from scratch. The first time around took the Age of Enlightenment and the Age of Revolution.
It's perhaps time to do that.
Prove me wrong. Seriously. If there are incremental solutions which stop deepfake porn, deepfake scams, personal drone strikes, and super-viruses, please let me know what they are.