Synthesizing Obama: Learning Lip Sync from Audio | Hacker News Reader

Synthesizing Obama: Learning Lip Sync from Audio

grail.cs.washington.edu

50 pointstaytus25 comments

Does anyone remember about 10 years ago, maybe 15, there was a guy somewhere in Europe that created something similar, as his graduate or doctoral thesis. He demonstrated it by taking the video of Michael Jackson denying sexual abuse allegations. However, he, the creator, was speaking into the system, and the video and audio of Michael Jackson "saying" the same thing was playing in near real time.

He explained his system at a fairly detailed level for a short 5 minute video. He was breaking down the source clip into very small snippets of time, and with some algorithm, giving each snippet a single value, or a score. Then, as his realtime audio was analyzed in the same manner, the source snippet with the closest score to the realtime snippet was output, both audio and video. I've always wondered what happened to that project, and I've since forgotten the creator or the name of his project.

Scrambled Hackz! I also remember running across this video.

So I shared the video with a friend yesterday. To my surprise they were not convinced. Which makes me wonder if it breaks down for very adept lip readers.

It doesn’t pick quite the right movement for some of the subtler sounds. I’m still trying to quantify where it breaks down for me.

It seems almost off from the audio. I know it isn’t but it just feels that way. Also the lack of eye movement and a few other “natural” things throw it off a bit for me.

For me, I think it's that the movement of the rest of the mouth and the muscles/skin in the face (and the light/shadow as a result) doesn't match up with what I expect. It looks like the facial animation from a good videogame cutscene

The future is scary.

* deepfake can superimpose a face on another face

* Adobe is developing a program to make a voice say anything (from a couple of hours of voice input)

* this can sync you mouth to the new voice generated by adobe.

Combine all three.

Eh, it won't be so bad. People are already accustomed to the fact that text, handwriting, and images can be easily faked. Voices can already be convincingly faked with practice prior to software assistance and there are lots of classes of videos can be faked with effort like security footage.

People aren't dumb and it'll only take a few good fakes to get people to distrust videos not attached to a reputable source.

doesn't all the controversy around fake news support the exact opposite of your claims that people are smart enough to only listen to reputable sources?

I would love to hear a digital forensics expert weigh in on the doomsday prophesies surrounding this sort of thing. Yes,digital fakery is going to get more and more convincing, but we are not without weapons in the defense of truth. http://ieeexplore.ieee.org/abstract/document/5487389/

I don't think this will work in the long term. Adversarial learning should learn any differences.

Here we go. The end of believing anything you see.

So— there is a way to defeat this, which is to track down the original source video that was modified to match, but yeah, in a few years this is going to wreak havoc on news reporting, intelligence, and the justice system.

People are going to have their lives destroyed by intelligence services over this.

>People are going to have their lives destroyed by intelligence services over this.

there's a paradoxical effect to it though. If these technologies become so pervasive that they're not distinguishable from real footage they'll lose their entire value, especially in a legal case. If you could arbitrarily create fake dna tests there'd be no point to pay attention to the results.

You’d think so wouldn’t you. AFAIK some courts still accept lie detectors as testimony.

I didn't actually know that countries are still using them. I thought they were a thing out of 80s American crime films. Consider me surprised

I think they are quite common in Israel.

https://www.timesofisrael.com/when-a-job-interview-turns-int...

> In Israel, too, the results of a polygraph test are inadmissible in criminal court but in civil court they are fair game if the person being tested agrees to it in advance.

Every(?) court accepts eyewitness testimony which is being recognized more and more as being very unreliable. Even ignoring the problem with eyewitnesses lying (or jurors falsely believing they are lying), memory is much less reliable than people feel that it is. It's very easy for false information to become associated with a memory where it resides forever after. Additionally, when remembering something the brain is happy to take the essence of something and fill in the specifics - it's as much imagination as it is recall - and it happens so fluidly it's hard to tell what you actually remember.

For an illustration of the second point, try to recall your earliest memory. For me, the first time someone asked me that question I had an answer which was then fixed in my brain as "my first memory". If I try to recall the memory, it's very apparent that there's nothing real left. It's just a memory of a memory, and any specifics are imagined extrapolations of home video and photos which I can actually remember. When this happens for more recent memories, the process is much more transparent.

Could you prime juries for unfriendly imagination by showing them inadmissible "simulations" of the crime?

If a judge catches you doing this, i.e. it wasn't an honest mistake, you might find yourself in front of a new jury.

Check out [1] for a discussion of current 'allowable' priming practices. Courts and legal scholars aren't blind to these tricks.

[1] https://ilr.law.uiowa.edu/print/volume-101-issue-2/the-inves...

>So— there is a way to defeat this, which is to track down the original source video that was modified to match,

Use an look-alike actor to make your own video. Then faceswap with deepfake. Sound can be taken from Adobe Voco.

But you need a thousands of pictures and hours of audio for this to work. And it's yet probably stil recognizable. But in a few year, this can be quite scary.

Signing and authenticated chains of custody will be necessary. That'll work in legal and other expert contexts. But yes, it'll be hard for most individuals.

Relevant Radiolab episode:

http://www.radiolab.org/story/breaking-news/

http://futureoffakenews.com/videos.html

I immediately thought about that episode as well. We need to prepare for the future of fake news on steroids when this gets even better. It's really scary.

This is scary.

Reply on news.ycombinator.com