A “deepfake” is at the center of a harassment case, but what if it’s not faked?
dailydot.com
dailydot.com
1. A cheerleader's mom anonymously texted some images and a video of a rival cheerleader to her coach. One of these images was a deepfake—a swimsuit photo off of Facebook edited to make the subject look nude. The other images and the video (of the girls drinking and smoking), were real.
2. The girl and her mom were charged with cyber harassment and harassment. The prosecutors weren't entirely clear on which of the photos + video were real and which were edited—the victim claimed they were all fake—but the whole thing was generally classed as harassment. Later, the police claimed that "metadata analysis" proved the video was fake.
3. ABC (among others, but ABC is mentioned in this article) ran a TV spot about this story, repeating the prosecutor's assertion that the video was manipulated
4. Twitter user @HenryAjder ("Deepfake expert"), among others, pointed out the error
5. The DailyDot published this article
This seems like a bit of a nothingburger? Any reason we're still talking about this? Nobody was "discrediting evidence", the videos & photos were only evidence to one thing—that someone sent them to her coach in order to get her kicked off the team. Whether they were real or not was immaterial to the harassment that occurred.
Nations state > Bigco > Smallco > Neckbeards > Karen
(skipping some steps but you catch the drift).
The metadata stuff the NSA was doing 20yr ago is what advertisers are doing today. SpaceX is doing stuff that was recently only the purview of nation states.
> Weintraub also stated during the press conference that investigators determined that the vaping video had been manipulated after analyzing its “metadata,” a term which refers to embedded information in digital media that can reveal how and when it was last edited.
> But after contacting Reiss, the officer who investigated Spone, the Daily Dot learned that police never actually obtained the original vaping video. Instead, like what was seen on NBC News, police only had access to a cellphone recording taken by Spone of the vaping video being played on a separate device. Any metadata analysis would therefore fail to include information on the source video.
It appears the police are calling the video of the vaping a deepfake
what was a takeaway here regardless of the story being wrong, is that the damage deepfakes will cause might not be what self-proclaimed futurists predicted (e.g. rampant use of the technology as a means to subvert evidence) but the opposite: that people will counter justified allegations of evidence as being deepfakes. So not the technology is doing most of the damage but the hypothetical possibility that allows that these arguments now exist.
Imagine any Karen or Kyle now being presented with video evidence will soon scream "deepfake". And few have seen this coming because we were more absorbed with hypothetical (but less realistic) scenarios.
This is even more fascinating (and was the simpler answer yet most have missed it).
https://cla.purdue.edu/academic/english/theory/postmodernism...
That's not the opposite, that is subverting evidence.
Also, please stop using ordinary names as slurs.
Currently we have reached peak character assasination. And you can ruin a life at this point because of one moment caught on tape.
Who cares if Karen or Kyle did x,y,z. Specially if it was intended to be private.
The rising millennial generation is not ready for the chaos their progeny is about to unleash on the world.
This.
I have been in a jury, and I wouldn’t be surprised if image based evidence won’t always be discounted by defense attorneys. It probably is already starting for the high paid ones (e.g. the notion itself will protect the rich.)
It brings up an interesting future where you can’t trust any witness testimony (because people do misremember) or any video, image, or audio evidence due to this technology.
The only societal workaround will be to implant cryptographic keys inside of bodies and monitor all interactions on some blockchain /s?
I think the danger is in trying to come up with an abstract, general solution for it. I don't think one exists.
I think the only thing we can do is deal with things on a case by case basis in these situations.
This kind of "verification" doesn't work. You're trying to have the device authenticate the image. That implies a root of trust that goes through the device manufacturer, which state actors will immediately compromise, making the signatures untrustworthy regardless. Then people will find vulnerabilities in the devices themselves, or convincing ways to have the device take a picture of a doctored photograph instead of a picture of the world, extending the ability to forge signatures to the general public.
The resources needed to forge a signature aren't going to be much different than the resources needed to create a deepfake to begin with. The creator gets to choose which device to use and can choose based on which device they find a vulnerability in. So it's more likely to be a liability than a benefit because it lends false credence to the authenticity of fakes.
For it to work you would need all devices to be secure against all attackers. It's not realistic.
How? If you have a public/private key pair and then do some kind of multisig process wherein the manufacturer can sign the photo, the OS can sign the photo, and then a user supplied signature could also sign the photo. You're not going to be able to fake 3 cryptographic signatures without 3 private keys. It would be extremely difficult to fake all of that. It would take a concerted, concentrated effort and would be extremely rare in practice. It would certainly improve the situation.
This image can now be injected into the news cycle and be taken seriously.
Keeping a secret on the camera is more or less the "jailbreak problem", aka being able to make a dvd player or console that only does the things that the manufacturer wants.
I will be pretty sad if this problem is actually "solved" effectively because it portends a future where manufacturers can completely curtail user freedom. But...
If the key on the device is compromised by reverse engineering or because the manufacturer is compromised then the whole root of trust is gone.
Attackers can choose the weakest brand of camera.
You now have some doubt about any image that has been "verified" because it could have come from a compromised camera secret.
All the rest of the stuff, the blockchain, etc depends on the root of trust in the camera.
You also now have debates around camera country of origin because there is no near future world where governments can't lean on local manufacturers. Finally, doubt can be thrown on images retro-actively if we later learn that a camera we thought was secure, has vulnerabilities.
All of this would be optional, of course. People can always edit out metadata. But it would lend credence and authority to a photograph or video. Which would carry more weight in a court of law, for example, where this stuff really matters.
I think the disconnect might be that you are working from a threat model where someone else is trying to tamper with an image from your camera. I agree you could make that arbitrarily difficult. But I am primarily concerned with the threat where someone is tampering with their own camera to fake evidence. This possibility makes it harder for you to prove that your own image is legitimate.
Your reasoning implies that one or two hacked cameras are not a problem. To my way of thinking, the possibility that "at least 1 hacked camera exists" or "any camera could possibly be hacked" are enough to throw doubt on any "verified" images that someone tries to present in a high stakes context such as national security or the courtroom. We're back to debating provenance.
It would require a larger degree of premeditation and effort to convincingly pull off a deep fake in that world.
It's not a technicality if the video can't be distinguished from perfect nonsense. Just as you don't let someone with a documented history of hallucinations testify on their own.
Or, you'd adapt a hardware wallet to do the same. Either way, there'd be two devices from two manufacturers, communicating over a transparent protocol.
I imagine deepfakes will become way easier in the near future because there are legitimate commercial reasons to make them easy and accessible (for example digital art assets). I don't see the same happening for cracking devices, which is more a standard security cat and mouse game between hackers and device manufacturers.
If someone presents altered footage, you can show that the checksum of the original was uploaded earlier.
This of course only covers certain cases. If the deepfake is crafted from scratch or from a camera that doesn't upload its checksums, and the incentive for the creation of the deepfake arises before the supposed time the deepfake takes place, then this method won't be of use.
And subject to attack. Don't want to be recorded? Jam wireless so they can't have anyone sign their videos, then claim they're fake.
Also doesn't work for state actors. A country can compromise three separate signing authorities.
And you can just upload it to ten different authorities in ten different jurisdictions, or even a hundred if you're paranoid. States are powerful but they're not _that_ powerful. It's the same principle that makes Tor work.
I agree jamming wireless could be an issue, but only for prosecution. In the case of proving innocence it's a moot point because this checksum system only covers after-the-fact fabrication anyway.
They'd also have logged the IP, which combined with a time will probably significantly narrow down the individual in question if you can also access telco records. Especially if inadvertently more information is also documented (such as a user agent).
> It's the same principle that makes Tor work.
Tor doesn't work for this kind of problem, unless you are significantly more technically savvy (and even then...) then the average smartphone user.
Yes, if a state actor wants to create evidence to frame you, and they didn't care about cost or returns, they could do so. But they could always do so. The fact is most of us are never going to get framed by a nation state.
The notion that you're either secure against all possible attacks or nothing is absurd.
> Yes, if a state actor wants to create evidence to frame you, and they didn't care about cost or returns, they could do so. But they could always do so. The fact is most of us are never going to get framed by a nation state.
This is a major problem for elections. Nation states absolutely will doctor video of adversarial political candidates and for other propaganda purposes.
> The notion that you're either secure against all possible attacks or nothing is absurd.
But that's how signatures work. You're trusting every device in the world to sign the output of its camera. If the attacker can compromise any device, they can produce signed forgeries.
Again, nation states have always been able to doctor video of adversarial political candidates. Lookup pictures where the KGB made someone disappear.
Yes, if an attacker can compromise any device they can produce a signed forgery. But that's a really big if. There are a lot of potential attackers who can't compromise any and all devices, and signatures protect against them.
Scaling the image down seems like one of the best tricks if you can get away with it. Another cheap one would be shifting the image 4 pixels over to break up the DCT artifact grid.
With a good forgery it will be. Even then, the fact that it's noise means that you won't ever be sure about this.
Just saying "we need to sign photos" isn't enough and simply naïve.
Are you saying that by law devices should cryptographically sign pictures?
The reality is that for most of those tens of thousands of years justice systems were mostly a sham where the focal point wasn't actually determining innocence but rather the appearance of justice and pleasing (some) people.
Our justice system worked for a very long time in an environment where cell phones didn't exist, where cities weren't covered end-to-end in CCTV cameras, and our courts were able to do a decent job of protecting the innocent and putting away the bad guys. Distrust of video evidence only makes us revert back 30 years to when we weren't so monitored. It's something that we can survive. On the other hand, if we don't see an outright rejection of deepfakes in our courts, we're going to see it used as a political weapon. We can survive the former, but not the latter.
Unfortunately, I don't see it working out that way. Once courts and police get their hands on a new technology and it becomes sufficiently ingrained, it doesn't matter how little sense it makes. Drug dogs are bullshit, but they're still everywhere. [1] DNA evidence can be synthesized and planted, but it's the gold standard in terms of courtroom evidence. [2]
Given all of this, I don't have a lot of faith in our justice system getting deepfakes right, and that worries me a lot more than guilty people being presumed innocent.
[1]: https://www.washingtonpost.com/opinions/2019/02/05/supreme-c...
[2]: Good reason to avoid giving your DNA to 23andMe or to the state. Unfortunately, both my parents did 23andMe, so I'm screwed regardless. https://www.nytimes.com/2009/08/18/science/18dna.html
Actually, it probably is somewhat. I hate paying money but that my taxes go to useful things makes it bearable. I hate sucking up to a guy in a uniform but that he also arrests murderers makes it less infuriating.
But moreover, that's not the issue brought up originally, which was net damage to a group. The implication was that the justice system harmed minorities and as a group that's simply not true.
Also, the language used is trying to borrow outrage. Harmed minorities? No, harmed the poor. Many of whom were minorities. But there are rich racial minorities too.
Per the Innocence Project, 70% of the cases they have exonerated have been of a minority group, and often these cases involved underlying racial prejudice to railroad an innocent person into a long-term prison sentence or death.[2]
>As of November 2019, 367 people previously convicted of serious crimes in the United States had been exonerated by DNA testing since 1989, 21 of whom had been sentenced to death.[9] Almost all (99%) of the wrongful convictions were males,[19] with minority groups constituting approximately 70% (61% African American and 8% Latino).
1.https://www.sentencingproject.org/publications/un-report-on-...
2.https://en.wikipedia.org/wiki/Innocence_Project#Overturned_c...
If the problem is racism, in a country where sexism against women is said to be a major problem, why would there be a 99:1 preponderance of men? Are the racists also biased in favor of women and against men?
Are males a minority group? No, technically not, since sex split is about 50/50. Is the high preponderance of men ending up in prison concerning? Yes. Frankly our high incarceration rates for non-violent crimes is extremely concerning. Does this somehow invalidate the problems brought up around female equality? Absolutely not. I also believe there is probably additional negative bias towards African-American males because of racist tropes.
Also the 99% figure was from the Innocence Project, Google brings up Federal Bureau of Prisons stat suggesting active incarceration is closer to 93% male, 7% female. Still concerning.
Hasn't anyone told you that you learn more with questions than assertions?
No. I'm saying that the language used is throwing out the baby with the bathwater. Regardless of all the stuff you mentioned, which is true, the system is still a benefit.
You should not go out of your way to further weaken trust in the system that has benefitted those minorities you mention more than it has hurt them. Especially as you are presumably not those groups, you should be careful not to wreck what they have. (You may not be aware, but white 'progressives' often speak for people of color. Their messages sound like ones of support initially, but because these people are often merely social signaling the rhetoric can often prove harmful to people who have to live with it.)
> [...] all disproportionately impact minority groups compared to Caucasians
Some minorities, yes. Others, no. Deeply troubling to the white v black narrative is that Nigerian immigrants often do very well in the USA, even when Americans don't know if they're american-descendants-of-slavery or not.
But yes, ADoS do have it rough. If you want to support someone though, vague "minorities" is not how you do it.
> Per the Innocence Project, 70% of the cases
You can't use that stat in that way, presumably they picked the most egregious cases which would be the poorest, etc.
>You can't use that stat in that way, presumably they picked the most egregious cases which would be the poorest, etc.
No, you cannot just ignore ingrained racism within the judicial system from day one as simply being a class issue. You can read more from the National Registry of Exonerations about racial bias in exonerated cases[1].
1. http://www.law.umich.edu/special/exoneration/Documents/Race_...
Just because some abuse them and have no proper training doesn't mean they are fake. You clearly have never seen a police dog work. They are able to find dead people, drugs, etc. that is at the bottom of the sea or follow the path the target took while inside a car. What is BS is the "police" and justice system in the US.
Yes, dogs are capable of sniffing out people, animals, and substances. This has been made use of for centuries.
Dogs are also capable of "hitting" on the trunk of a car, just because they guess that their master wants to look inside it. Plenty of evidence that this happens all the time.
Which undermines the right protecting against unreasonable search and seizure. All you have to do is be the wrong color, or driving in a "suspicious" way (which means whatever police want it to mean), and they can bring a K9 by, the dog will dutifully point at the trunk, and the cops get their search.
It's no different from the "I smelled pot" routine: there aren't any penalties to the police for doing an "I smelled pot" search and not finding cannabis, and there are no consequences for a K9 'hitting' on a trunk which turns out to have nothing more interesting than a gym bag.
That future sounds a lot like the past where we couldn't trust testimony and audiovisual evidence was practically nonexistent.
I remember that TruePic was especially awful, because their app fell for me just taking a photo of a printed out image. Plus their website overflows with Blockchain and Crypto buzzwords
Not to mention adversarial attacks will probably always exist, so there will always be a way to fool these systems.
The prosecution projected a film with audio [cutting edge technology at the time] demonstrating the crimes in question [bribery and related charges].
Lamont ['The Shadow'] was able to hear the audio and read lips at the same time and figure out that what was heard was not was what said. "Believe half of what you see, and nothing you hear" was his takeaway.
As the story goes, that's what happened. The film shown at court was dubbed for audio by a hired guy who did great vocal impersonations. The accused was innocent.
Short story long: deep fakes aren't a new idea. 100 years and counting. Same trick, different tools.
If its really fake the folks over at Sassy Justice should hire her, their stuff looks like garbage in comparison.
https://www.dpreview.com/news/7021408195/hipster-offended-af...
But after contacting Reiss, the officer who investigated Spone, the Daily Dot learned that police never actually obtained the original vaping video. Instead, like what was seen on NBC News, police only had access to a cellphone recording taken by Spone of the vaping video being played on a separate device. Any metadata analysis would therefore fail to include information on the source video.
In response to questions on how it could have been determined that the vaping footage was manipulated without access to the original video, Reiss argued that he could see with his “naked eye” elements that “don’t make sense.”
-----
So... They can tell by the metadata on a copy of the file plus a naked eye analysis of the video that this is a deep fake? That's much worse evidence for the deep fake case than I assumed there would be.
This does make me wonder though: Should we default to believing video or doubting it? Does the victim here need to prove it is a deep fake or the perpetrator prove it's not?
Never expected to see a version of the "I can tell from some of the pixels and from seeing quite a few shops in my time." meme used as legal testimony.
You just get an expert witness (or opposing ones) to testify as to the likelihood of something being forgery or not and why, as well as take things like motivation, means, etc. into account.
There's literally nothing different about deepfakes. We have a well-established court system for handling this.
That's a bit disingenuous don't you think?
No. What's disingenuous about it?
From the legal perspective, I think you're right that "at the end of the day" faked evidence is faked evidence and we do have protocols and processes for that kind of thing. Dismissing faked evidence is not new.
However, earlier you said this:
> There's literally nothing different about deepfakes.
I'd argue requiring AI/ML to create or invalidate deep fakes is _literally_ very different from generating fake emails, fake photos, or fake audio. It's also _literally_ different from things like CGI video. The standard of what counts as evidence is _literally_ changing as deep fakes evolve.
Validating video evidence, in the new era of deep fakes is now very different.
Your clarification about the context of the courts ability to deal with them is an example of how you were being disingenuous, in my opinion. You made a blanket statement about deep fakes are literally the same as other faked evidence and then afterwards put a clarifying condition around the statement to narrow it to the court's ability to deal with faked evidence.
If we're just talking about faked evidence and how to deal with it - I think we agree: Determine if evidence is legitimate however you have to, and then toss it out if it's not legitimate.
But we're talking about deep fakes in this context and your statements dismissed all of the things that make deep fakes different in the context of determining legitimate evidence, or how deep fakes will impact the court system. We are transitioning (over decades, already) from a standard of evidence of "This is video evidence of the truth of what happened" to "Some people think this video is probably not real." That's a big difference.
> The statement is similar to claims made by Engelhart to the Daily Dot that Spone allegedly used legitimate video as a basis for the deepfake. Police did not know whether the alleged victim’s face was added to a video of another young female vaping or if a vaping pen and smoke was digitally added to a legitimate video of the alleged victim.
If you're saying that she started with one video or image that was publicly accessible and modified it, I'd expect them to have the original video or image so they could tell what was added.
They may be able to generate a "plausible" nude, suitable for rubbing one out. But it's not a real nude. Moles and scars wont match. The data is just not there, so the manipulator has to make something up. Thus I think the nudes are actually the most promising for conclusively proving or disproving a manipulation.
This happens.
Particularly in rape cases where the woman (or child) describes some particular feature on the alleged rapists genitals.
Michael Jackson had to strip down, as I recall. Obviously not in open court.
You’d spend a lot of time and effort explaining one mole difference - and that’s if you legitimately had one. If you didn’t, what else are you going to go on?
but I suppose it will follow the same standard as sexual harassment and rape victims: if they’ve ever altered the truth in the past then their current experience is invalidated for the purpose of using a court to reduce freedom for someone else
so videos will have to be closest to raw sensor data or inadmissable
You could also certainly get an expert witness to give their opinion on whether that video is CGI or not.
To lay the foundation for admission for a photograph into evidence, ask questions such as these:
Q. Mr. Witness, I'm handing you a photograph that's been marked Exhibit 2 for identification. What is depicted in that photograph?
A. A stoplight at the intersection of 4th and Pine.
Q. Is that photograph a fair and accurate representation of the stoplight at 4th and Pine as it existed on the day of the collision?
A. Yes, it is.
At this point, you can move for admission of the photograph into evidence.
https://www.illinoistrialpractice.com/2004/11/foundational_q...
On cross examination the other side has the right to try to discredit the witness: “Aren’t you really blind, couldn’t this be a deepfake, “ etc. And, they have the right to put on an expert or other witness to testify that it’s fake or the wrong picture or whatever. In the end, it all goes to the jury and they get to make of it what they will.
When CNN starts using deep fake tech to show Trump throwing a bowl of fish food into a pond, then it's reached maturity.