Deepfakes Tore a High School Apart
404media.co
404media.co
Cameras cryptographically signing all photos and videos should become the standard. Media that cannot be traced back to real camera hardware should a priori be considered fake or tampered with.
Still, deepfakes are partly a social problem and, as should be well known, social problems can rarely be solved by technological solutions.
I think you meant to say: a single eastern European teenager who gets bored one weekend. These systems have been around for decades, and they've continually been hacked for decades too.
How about we undo the radical individualism that has rotted out any compassion and empathy from society in the pursuit of greed? Not as easy to do granted, but it's a social technical debt that can't be quick fixed.
You sound like you've given up on nurturing humanity and instead want to enforce good behaviour as if the base human condition is to hurt others by default.
These are often young boys behaving in ways that are misogynistic and heartless. Usually following the examples of heartless parents and peers. We can do better. They're kids learning the wrongs lessons about the world.
The modern web has not been without its warts. It's led society to some very unhealthy behaviors. It's telling that a lot of people think the internet is society, rather than a slice of it. Perhaps its decline is the correction society needed.
It's easy to feel like a doomer if you're thinking "it's sad that the web is being torn apart" and your framing of the web is "the magical place of promise it used to be 25 years ago."
Funny that cd's are comparatively analog. (I get your meaning though).
It’s already been tried, when deepfakes weren’t even that good.
https://www.theguardian.com/technology/2023/apr/27/elon-musk...
But then again, perhaps we won't and things will just get worse.
The older generation (gen-x) and before has been trained to believe traditional media (and by extension, anything that is published on a "website" and/or forwarded on Whatsapp). Images, pictures etc. add to the authenticity. Of course, it's easier to forward than to investigate and refute.
The younger generation is less influenced by this but also care less about generating and forwarding these. Most of these things are "just a joke". In a weird way, pushing out non-consensual nude of someone is the same as editing someone's face in a group photo with dog ears and sending it to a circle of friends.
I don't know what the solution is but this kind of thing erodes trust. Photographs used to be evidence that can be used to establish trust and we've been culturally conditioned to accept them that way. This takes a sledgehammer to that and it's not easy to untrain a whole society away from a deeply conditioned feeling so easily.
I'm just waiting (with some dread) for the day that you can sync up multiple cameras with AI to fake from different angles.
You will have no compute and you will like it. The top comment here is about arresting the children for distributing child porn.
We really should have kept the normies out.
"make the person in this photo naked. No mistakes"
Prompts lowered the bar. Even if you do have to install a local text to image model.
there are, for example, websites that host (among a myriad other such things) style-accurate pictures of every Disney princess being fucked in every way imaginable. 70 years ago something like this might have caused a similar moral panic, but now no one (rightfully) gives a shit. you see a picture of Cinderella with a dick up her ass, you know it wasn't drawn by Disney.
Theyre not the same thing and you can't just hand wave away people's pain. You're part of the problem.
Since perfect or not people aren't going to stop doing what they do, the fact that the fakes have gotten better is only beneficial since it allows for plausible deniability. At this point nobody should trust some random video the same way nobody has been able to trust a random JPG since photoshop became a thing.
Yes, from a victim impact perspective.
> Before computers people would cut out peoples faces and paste them onto playboy centerfolds. That sure wasn't perfect, but it didn't stop them
The effort did stop them. The problem wasn't even half as widespread. And those who were doing it were keeping it to themselves, the images weren't being auto-shared to Twitter and the targets tagged.
> the fact that the fakes have gotten better is only beneficial
[Victim testimony needed]
> At this point nobody should trust some random video
While there's a qualitative difference in harassment when using media that is either real or fake (clearly or not) the capacity for that media to be used for harassment purposes remains the same.
I hope we don't start getting xkcd fakes.
The PD’s PR team changing their narrative seems to lack competence. The school board focusing on being light touch on a specific case instead of cracking down in general seems like another error of judgment. This whole article says more about the effectiveness and competency of cops and school boards as it does about generative imagery.
I skimmed but the worst I saw was one of them referencing her as “that hoe”. Not nice but also punishing kid’s private conversations outside of school this closely seems like an overreach. Worse, is the message it teaches the younger generation that speech should be censored and isn’t as free as they think. I feel that’s a worse outcome on a longer timeline.
The framing bothers me. It's shifting responsibility away from kids and the community. Deepfakes didn't do anything: kids made and used deepfakes in destructive ways.
Big tech sucks, you get no argument from me there. But, how do people so quickly get over the fact that a kid is sexually harrassing someone?
This is like when a mass shooting happens and everyone only talks about guns. Well, yeah, if you didn't have guns a mass shooting couldn't happen, but isn't the underlying issue that someone decided that hurting a bunch of people is the best thing to do?
The more people dislike a crime, the less likely they are to talk about the perpetrator's inner life, but, come on, these are kids. How do people get so side-tracked?
One of the mothers says this quite explicitly:
> Calling the event rumors and speculation when a crime occurred was almost worse than the crime
What's the defense? Intelligent screening of incoming messages so that the threat never reaches the blackmail target? I imagine they'll find an unprotected channel.
Don't post innocuous images of children ever? Seems like losing.
I do have to say that I find it disturbing how liberally parents seem to post pictures of their children in public though. Respect the privacy of your children, they’re not your pets.
Such things were printed and handed out and rarely made it past that years parents and students and the school archive (physical visit required).
For the Dux's and general high performers, less time than I've been alive.
For entire classrooms to be posted up via live streams, Insta, tok's, etc ... barely a decade or so.
It's not something that was always normal, and just because it's become "normalised" doesn't mean it must remain that way.
I think a school news letter / yearbook is cute, but it should be private and only for the students and parents. A physical version would be best, you can put those in the archive instead of realize 20 years later that the digital versions are gone.
Saying "you shouldn't do this thing that was basically harmless from the birth of photography until the ~2010's" is ignoring a lot of history and context.
Prosecution of AI operators for making indecent images of children?
Hackers and blackmailers on the internet can be anywhere from trivial to impossible to identify.
Even harder to prosecute when they're in a different jurisdiction, as not everywhere has extradition treaties to everywhere else. I wouldn't be overly surprised if e.g. the North Korean government runs some schemes like this (though not necessarily actually this) to bring in money.
But yes, definitely do the easy things first, like trying to stop e.g. grok from doing it and also going after users of e.g. grok who try to get around such efforts. A lot more crime happens than can be prosecuted, so raising the minimum competence threshold to commit crimes in the first place is very necessary.
Or to elect me as god-emperor and unify the world under my rule.
Same defenses that are used against fraud and other crime.
criminal prosecution of the blackmailers AND the services used to generate the pictures.
This is effectively child porn... so penalties would be pretty harsh.
There are extradition treaties to most of the world, so unless the blackmailers are in China/Russia they will end up in jail.
That same thing played out with piracy with people extradited to the US from various countries
As an aside, please do not use the b-mail word. It is insensitive towards BIPOCs.
No it isn't. If anything needs to stop its the ever escalating orwellian censorship of words that you're proposing.
Model makers, arguably, are like pencil manufacturers but in a World of good artists.
I think you can hold people to account for what they help to create, but not what they have potential to do.
Now, if models were trained on any csam, of course, the model's owners should be held you account.
I still think there's a lot of legal and IP landmines lying in wait from the Hoover-esque pre-training eras
Agreed on your second remark. If I'm still around it will be interesting to look back on this period on twenty years and think about how things should have been handled.
Don’t raise boys who abuse girls. That’d be a big step in the right direction.
[0] https://www.abs.gov.au/media-centre/media-releases/97-cent-s...
Prosecuting every one of those blackmailers. If this kind of crime starts going into the "you're gonna get caught and jailed 100%" category, less people will try it.
You won't prevent people stabbing others with knife by banning knife or asking people to wear knife-proof vest going outside. You deter them by making everybody know that the consequence of harming somebody else is going to be a very unpleasant experience.
There’s a massive difference.
(And by the way the implicit comparison here between an image of stalin and an image of a teenage girl is more apt than most would care to admit given the ideological function fulfilled by sexualized images of young women in contemporary American society)
What is wrong with you?
It wouldn't be the first time a high-schooler killed themself over bullying.
Something millions died to protect in the great patriotic war.
Again, symbolic violence is worse than real violence to a lot of people.
I don't have sex in front of my friends and family for a reason, and I would appreciate my privacy protected by the state. And yes, privacy breach of this magnitude is probably worth 2-3 years (which basically means nothing for the first offense, let's be realistic, but makes the second offense way more consequencial).
It's not like I can stop you from imagining it.
Adding a GPU to the mix does make it any worse given that you already thought about it.
A proper society should educate people about things they can't get rid of anymore.
And no you can't just put teenager, which have not fully formed brains yet, into prison for fake nudity
Teach them compassion and empathy. Teach them why their actions hurt.
Yet the consequences for the victims may last a lifetime and be both intrinsic and extrinsic. Sexually forcing themselves on a drunk colleague at a party or taking their dad’s gun and shooting up their classmates also only takes a few minutes, but can scar a community for years.
I’m not going to opine on what the punishment should be, though it seems clear there should be one and more than a slap on the wrist. The community where this took place (i.e. those who are living through it) should probably have a say in it. There should also probably be an investigation of how this came to pass, i.e. has this kid always been an asshole with a pattern of behaviour or have they recently been indoctrinated by manosphere bullshit and should get counselling to understand their actions and that women are people too and deserving of respect?
And there is a huge difference between clarifying with people what fake AI is vs. real nudity.
You can't just put a kid into prison because they copy pasted a head from someone onto some body.
Of course they should. I’m not saying they should get to unilaterally decide, but you should listen to the people who are affected. Don’t just ignore the people, the law is meant to serve them. Elected officials exist to represent the will of the constituents.
For example, let’s say you do something which is against the law but only affects the community you’re in and no one else, and the community not only doesn’t mind what you did, they are thankful. Should you be punished?
> A independent judge should do this to make sure its fair.
What do you think a judge does? They aren’t robots who only interpret a book, they listen to the people in a case and make a decision in context and within the framework of the law.
Furthermore, “independent judge” does in no way guarantee “fair”. Decades of interminably biased decisions aren’t going to stop now (definitely not with the current state of the US and its supreme court).
> You can't just put a kid into prison
Again, I’m not going to opine on what the punishment should be. I haven’t suggested a punishment, don’t argue as if I did.
> because they copy pasted a head from someone onto some body.
You’re misrepresenting what happened. The kid didn’t go in there with scissors and hastily pasted a head into a picture in a magazine. We know the state of generative AI, we can presume it was a competent fake.
My point was that this type of technology is so easy to access and to do, its not easy to just make it illegal.
Take a picture out of the year book, copy it onto a body, done
I don’t agree with the nanny state approach. America is already too obsessed with violence being ok and any kind of sex/nudity being completely evil. I don’t have an actual answer but having a moral panic over it is not the solution.
Worth considering that this isn’t “any kind”, it’s a very specific kind. It’s nudity of minors they personally know, shared in their community. It’s on a different level than, say, consensual nudity from a stranger in a Playboy magazine.
Surprise.
Am I getting this wrong?
Being curious about other people's bodies is part of the human experience. Sharing unwanted graphic depictions of them, is not the most mature act, but doesn't rise to the level extreme punishment as advocated by the GP.
> It used to be that if you wanted to see people naked you’d just use your imagination. If you really wanted you’d draw or use photoshop, but both actually required skill. AI made it too easy. Teenagers wanting to see each other naked is normal and part of the human experience.
Certainly there needs to be repercussions, the kids need to learn, and need to be warned in advance, but kids will do dumb things.
You can talk about news coverage, but you can't force 12 year olds to watch the news, or understand how their actions have consequences. When the consequences do come, the focus needs to be primary on rehabilitation and restitution for the first instance.
Redeemable children can make horrific mistakes. The job of parents, and pastoral adults is to maintain an environment where kids can exist without making irreparable, life-destroying mistakes. Let them learn, but limit the damage they can accidentally cajole themselves into doing.
That means limiting access to social media, phones and cameras. Turning the school clock back to 1995.
Otherwise this is insanity.
It's a disruption of an entire school and the learning process. Just like a bomb scare, or a fake fire alarm. Just because it's AI doesn't mean it gets a pass.
One of the points of punishment is to deter copycats.
I stand by my claim of insanity. I think your rebuttal says more about you than about the topic at hand.
Saying "don't make a fake picture of anybody naked or you'll go to jail and your life will be ruined forever" sure seems like a form of censorship
There’s really no way to close this Pandora’s box. So, I’d fight fire with fire instead of crying about it if I was a teen girl in this situation. Make deepfakes of the boys have micropenises and other potentially embarrassing things. And then just learn to laugh about it. If everyone is a target and victim the kids will find it boring fast enough and this will fizzle out. It’s not like these kids aren’t exposed to porn and all that early on. They all are, have been for years now, the victim here feel awful because they’re the only ones being targeted.
I know this sounds bat shit crazy but normalization of it is really all we can do now.
For example here the word "abuse" technically refers to the making and spreading of these images and attendant bullying, but obviously has connotations of actual sexual abuse:
>Despite all this, Radnor’s administration failed students in the days and weeks after it learned about the abuse,
This is maybe subtle but later the article gets more explicit about it:
>this technology [...] bears little difference from the non-consensual intimate imagery that’s plagued young girls and teenagers since the invention of the camera.
(i.e. a straightforward equation of images of nudity that did not happen, and images produced by actually pointing a camera at someone who is physically present with their nude body)
>Because the images aren’t “real,” authorities grapple with how to handle them.
Putting "real" in scare-quotes, very nice.
Dorfman is also quoted doing some nice sleight of hand, going in one sentence from "It was about the creation of the videos" to "and then share them with others as if the girls were something to be passed around." - i.e. equating the making of videos depicting unreal events with something that almost sounds like gang rape.
Anyway enough examples, article is full of them. I'm not saying there's no problem here, just that these people seem to have some kind of ideology of the sanctity of the image, i.e. a realistic image always depicts real events, and should always be interpreted as if it does. If it does not, clearly that's a problem to be corrected by reinforcing the connection between images and reality by making it impossible to produce realistic images of non-real things. It strikes me as exactly the wrong way to go about it, that can only make the bullying and harrassment (the actual problem which should be solved on the social level) worse by making its ammunition stronger.
(Slight aside but I do think the closing quote by Woelfel makes a good point)