Neural Network Enabled Filmmaking

But no matter the audience, cutting to the chase about what the product does is super important.

The criticisms here seem to break down into three groups:

1. "I don't mind reading subtitles and that will always be better."

2. "Any real film purist will prefer watching the original."

3. "The implementation is kind of uncanny at some points."

My thoughts:

#1: Sure, I actually agree but I also know that there are a lot of people out there that, for whatever reason, find reading subtitles breaks their immersion. Even though that's not me, dismissing that audience's experience isn't very inclusive.

#2: Yep, As a film maker I totally get that but us purists and aficionados aren't that big of a market. So yeah, Criterion should never use any tech like this for definitive remaster releases as it doesn't preserve artistic fidelity. But just because they should keep this away from Citizen Kane doesn't mean there aren't suitable classes of content. In the era of high-fidelity streaming media, additional versions don't necessarily take distribution availability from the original (which should always be available).

#3: Though I'm not an AI engineer I've done some work with AI engineers on artistic content and my guess is the current gaps we see might be sufficiently surmountable with refinement. It may never be fully automatic but redubbing is human intensive anyway so an interactive UI that allows an ADR-experienced editor to tune problematic output may help a lot.

I think the team should keep going. Once it's good enough, there is a lot of content that might reach new audiences through this tech.

I wonder how longer or shorter phrasing would affect clips or cut length and how backgrounds might also have to accommodate script line length.

This last year’s best picture went to Parasite, a “foreign” film with English subtitles.

While film people would probably always prefer the original, I bet most Americans would have watched Parasite dubbed with this technology and not even realized the actors didn’t use English in the creation.

Hollywood does butts-in-seats and I think this would put more butts in seats.

Also in your promo video you show “traditional dubbing” to English and I really expected that to be followed by your system on the same clip but instead you moved to other languages, which I can’t really grasp.

Edit: I see I’ve gotten a downvote. You can hear from the actor himself how he feels about playing that role: https://www.dailywire.com/news/fisher-stevens-regrets-playin...

To me, this is "good enough" to watch. I don't like dubs because the voice is out of sync with the character, and removes you from the story. I don't like subtitles because I can read faster than they talk, and then I'm left with words blocking part of the scene, and I feel like I know what the person has said, yet their still talking.

This is a great start, and they'll continue to improve in the future.

With things as they are now, I prefer original dialog with subtitles usually because the lip-sync is so bad, forced, and doesn't sound like the original actor. (Anyone remember how bad the sync was on some Spaghetti Westerns?) If this can do "deepfake" alternate language dialog using AI processing to change the language and the accent but in the same voice as the actor, I'm all for it. Although, the talent should get compensated because it's taking money out of their pockets if their jobs are being replaced digitally.

Given the quality of Google translate, “every language” sounds like they are overselling it.

That way, you can dub videos however you like without having to rewrite your dub text to match the visible mouth movements. I like the idea, but in my opinion finding good dub speakers is the challenge, not figuring out what you want them to say.