Hacker News Reader

Comment by sleepydog | Hacker News Reader

I noticed Netflix doing this while watching this series: https://about.netflix.com/news/beyond-goodbye-premieres-nove...

It was so jarring I couldn't watch it and switched to Japanese with subtitles. Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.

View on HN

You can see a lot of open mouth/tongue stuff being skipped. Dealing with the tongue and the inside of the mouth is a huge problem with this sort of visual dubbing. Using traditional techniques, you can model teeth and gums as rigid bodies, and faces as rubber sheets (to first approximation), but tongues, for which you typically have no visual reference in any given shot, are much more difficult to model, and continuously, subtly, on the move. "AI" is the general answer to this problem nowadays, but even ML-based systems struggle to deal with the tongue issue while trying to reconcile visual appearance with animation fidelity.

But the actors re-recorded their dialog in English in the studio. Why wouldn’t you video record their mouths? In fact, the old punch the mouth out and pop the new one in behind the whole can be surprisingly effective sometimes. I would think having an actual visual reference and doing more of a deepfake on the lips should provide excellent results, vs synthetic lip motion to match the audio only.

Yes, Kung Pow is a great example of how jarring it can be! :-D

"YOU WANT FRIES WITH THAT?"

It's funny how older movies like that are unintentionally prescient.

Like the Kentucky Friend Movie "Eyewitness News" segment and smart TVs.

> Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.

Everything old is new again. Classic Italian films were almost always dubbed by their original actors, to the point that Fellini sometimes didn't even write portions of the dialog till after filming was done. The actors just said random stuff during filming. Originally this was mainly a technology limitation (sets were too noisy), then it became a cost-savings measure and cultural tradition to how films were made in Italy.

I always find it a bit of a distraction when I haven't watched an Italian film in a while, but eventually get used to it.

Dubbing by the original actors is very common when making movies. But usually the dialog is the exact same so it’s not noticed

> But usually the dialog is the exact same so it’s not noticed

That's not the case for the films I'm describing. Here's the start of La Strada:

https://www.youtube.com/watch?v=7jSewbxyHYM

Quinn isn't even speaking Italian during the shooting:

As was the common practice for Italian films at the time, shooting was done without sound; dialogue was added later along with music and sound effects. As a consequence, cast members generally spoke in their native language during filming: Quinn and Basehart in English, Masina and the others in Italian.

https://en.wikipedia.org/wiki/La_Strada#Sound

It's even more clear in this scene:

https://www.youtube.com/watch?v=f7fmvA_KG6U

Caligula et Messaline 1981 exist in two close but significantly different versions played by the same actors at the same time.

It was quite jarring in Amazon's Citadel Diana. The voices were ever so slightly out of sync with the lip movements, and the audio sounded like the studio recording hadn't been processed to match the environment.

Reply on news.ycombinator.com