AMC Theatres will screen a Swedish movie 'visually dubbed' with the help of AI | Hacker News Reader

AMC Theatres will screen a Swedish movie 'visually dubbed' with the help of AI

engadget.com

85 pointsbookofjoe101 comments

I noticed Netflix doing this while watching this series: https://about.netflix.com/news/beyond-goodbye-premieres-nove...

It was so jarring I couldn't watch it and switched to Japanese with subtitles. Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.

You can see a lot of open mouth/tongue stuff being skipped. Dealing with the tongue and the inside of the mouth is a huge problem with this sort of visual dubbing. Using traditional techniques, you can model teeth and gums as rigid bodies, and faces as rubber sheets (to first approximation), but tongues, for which you typically have no visual reference in any given shot, are much more difficult to model, and continuously, subtly, on the move. "AI" is the general answer to this problem nowadays, but even ML-based systems struggle to deal with the tongue issue while trying to reconcile visual appearance with animation fidelity.

But the actors re-recorded their dialog in English in the studio. Why wouldn’t you video record their mouths? In fact, the old punch the mouth out and pop the new one in behind the whole can be surprisingly effective sometimes. I would think having an actual visual reference and doing more of a deepfake on the lips should provide excellent results, vs synthetic lip motion to match the audio only.

Yes, Kung Pow is a great example of how jarring it can be! :-D

It's funny how older movies like that are unintentionally prescient.

Like the Kentucky Friend Movie "Eyewitness News" segment and smart TVs.

"YOU WANT FRIES WITH THAT?"

> Their mouths looked so stiff and unnatural and out of sync with their expressions that all me and my wife could talk about was how bad it looked.

Everything old is new again. Classic Italian films were almost always dubbed by their original actors, to the point that Fellini sometimes didn't even write portions of the dialog till after filming was done. The actors just said random stuff during filming. Originally this was mainly a technology limitation (sets were too noisy), then it became a cost-savings measure and cultural tradition to how films were made in Italy.

I always find it a bit of a distraction when I haven't watched an Italian film in a while, but eventually get used to it.

Dubbing by the original actors is very common when making movies. But usually the dialog is the exact same so it’s not noticed

> But usually the dialog is the exact same so it’s not noticed

That's not the case for the films I'm describing. Here's the start of La Strada:

https://www.youtube.com/watch?v=7jSewbxyHYM

Quinn isn't even speaking Italian during the shooting:

As was the common practice for Italian films at the time, shooting was done without sound; dialogue was added later along with music and sound effects. As a consequence, cast members generally spoke in their native language during filming: Quinn and Basehart in English, Masina and the others in Italian.

https://en.wikipedia.org/wiki/La_Strada#Sound

It's even more clear in this scene:

https://www.youtube.com/watch?v=f7fmvA_KG6U

Caligula et Messaline 1981 exist in two close but significantly different versions played by the same actors at the same time.

It was quite jarring in Amazon's Citadel Diana. The voices were ever so slightly out of sync with the lip movements, and the audio sounded like the studio recording hadn't been processed to match the environment.

On the one hand, as a lover of film, ewww gross.

On the other hand, as a lover of film, if this will bring more audience to foreign films, then let's give it a try.

This is very much in the same vein as colorizing black and white movies, pan-and-scan for video releases (and now zoom for 4:3 content on widescreen), and dubbing instead of subtitles. Every one of those things brought in more viewers of the content, exposing more people to those films.

I want to be on board with the “bring more audience” but the truth is knowing how this industry works, any success with AI on this front will lead almost immediately to a total drying up of funding for professionals engaging in this work currently. I hope I’m wrong, I really do.

What professionals are doing this work already? As I understand it, this is something that can only be done by AI (or by shooting the movie twice in different languages).

audio dubbing and voice acting is a thing. They even do it in a way that takes into account the mouth movement length/timing, to make it look slightly more natural a dub.

of course, it's not super good imho, and personally i prefer subtitles with the original audio. Even if ai did a perfect job, including changing the film's frames to suit etc, i dont believe the outcome is sufficient. I want the original actor's voice timbre and intonations, which has meaning, and is lost when translated to a different language.

This is using the original actor. They re-recorded it. The AI just modifies the lips.

From TFA:

> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.

Imo the real situation is that US distributors suck at dubbing and/or rely too much on celebrity voices rather than good voice actors that can dub properly.

I have seen some US dubs that would have been improved by a polish-style voiceover dialog

German dubs were (or still are – if an appropriate (time) budget is allocated) probably some of the best.

I dont get why anyone would want dubbing over subtitles?

There’s a reason ‘may your hobby hit mainstream audiences’ is an insult. Foreign films are foreign because they don’t attract a mainstream (read: anglophone) audience. As soon as there’s mainstream money, clueless producers will decide that they want that mainstream money, and make changes to accommodate said audience. This ends up turning them into a slightly different flavour of Hollywood.

Context: I think most recent use cases of AI is slop, generally people who identify as pro AI think I’m an AI doomer. But there are some good use cases of AI.

This sounds like a good use. It reminds me of the early days of CG, it took a while for it to look good, and now it’s a tool in the toolbox alongside practical effects. This is gonna look bad for years, but I think ultimately will be a good thing. Anything that helps shrink the world and understand other cultures is a plus for me.

But man, as a film lover I know this gonna annoy me over the next 5 years.

Agree on every point here, apart from your very last one. I am not watching movies with AI dubbing. Subtitles and original audio is fine. I do not speak a word of Japanese but I have never had issues at all with dubbed anime or films from that country. If this is a boon to anyone out there who finds foreign film utterly intolerable without it, genuinely, I'm happy that you will hopefully now have this option and I hope it's everything you need, but yeah. Hard pass for me thanks.

Cool, I also almost always prefer the subbed, but I meant a good thing for a wider audience for foreign films and TV shows. Imagine wider theatrical releases for, say, Let the Right One In, instead of a bad American remake. Think of possibly more money and recognition for the original creators. This is a good thing, even though the film lover in me is going to a visceral reaction against it.

Edit: I think I misread you and you actually agree with what I just said, so never mind. I have noticed with streaming more obscure movies sometimes dubbed is the only option, though

> I think I misread you and you actually agree with what I just said, so never mind.

Correct! I’m a very live and let live type of girl lol

If it looks good it's fine, but I'm not as optimistic as you.

I would argue that pan-and-scan failed long-term, perhaps even short term. I remember widescreen releases being available on VHS because a significant amount of people didn't like it.

Colorizing black and white movies...I can't think of any time I've seen that done in a commercially successful way.

I also think that a dub with lips that don't match is something people barely notice in the first place.

Did you watch the video posted in the article? Looks pretty good. The swedish actors recorded themselves speaking English and really the tech is just changing the lip movement. I think it would be more weird if they weren't speaking with a swedish accent.

I would argue that pan-and-scan solved a practical problem, and while it wasn't ideal, it was also just flat out necessary for a while. Your average TV in the '80s would have been around 20", and VHS only gave you 240 lines of vertical resolution. Letterboxing a 2.39:1 movie meant cutting both of those metrics in half. Preserving the original framing doesn't mean much when a sizable portion of your audience is stuck squinting at the screen.

Some filmmakers kept this in mind when making their movies in the '80s and '90s. Films like Terminator 2 were shot open matte, allowing each shot to be framed for 4:3 and the target theatrical widescreen aspect ratio simultaneously.

I think these facts got lost on a lot of people who were die-hard widescreen LaserDisc fans. LaserDisc had twice the vertical resolution of VHS. And if you were a big enough moviebuff to have a LaserDisc collection, you probably also had a bigger than average TV.

The rest of these techniques, like colorization, seem more about making older content more palatable to newer viewers, and don't really do anything to solve a technical problem.

It may have begun as a practical problem, but the fact that the opposite now happens (zoomed 4:3 content on widescreens which are all at least 720p and where letterboxing doesn't lose anything) tells me that there is enough of a population that doesn't like "the black lines" to warrant publishing those.

I wish they would just split the difference and zoom to 14:9. That minimizes the worst of the cropping with less intrusive bars.

David Simon went into great detail in his blog about why they had to completely re-edit The Wire scene-by-scene from source material to suit the HD format. There were scenes where the meaning was changed and details lost if they simply cropped without discretion.

https://davidsimon.com/the-wire-hd-with-videos/

By 2002 widescreen HD was clearly on its way. The video cameras weren't ubiquitous for shows shot on tape, but for film shoots, it's on them for not framing with awareness of a future HD release in syndication. The only excuse for composing shots in full frame 4:3 in the 00s would be retro-themed programs like That 70s Show.

I think you mean it's a technical problem. Language translation and screen size translation are both solving for the practical problem of reaching a wider audience and on more devices.

Subtitles and dubs already solve the practical problem of making content produced in one language accessible in another, and in a non-destructive manner.

This AI lipsyncing nonsense is more like the rash of colorized black and white movies that came out in the '80s, or the modern day scourge of 4:3 shows getting pan-and-scanned to 16:9. The "problems" these solve are viewers being closed-minded about watching movies and shows that don't have color or "look old". The final product is generally going to be inferior to the original.

My pedantry—like most pedantry—was focused on the problem, not the solution. Ergo, you could accurately say "AI lipsyncing nonsense" is a highly-technical solution to a practical problem, and subtitles a pragmatic low-tech solution.

I grew up watching foreign movies almost exclusively dubbed. French TV in the 1980s-2000s dubbed everything; it was nearly impossible to watch stuff with the original audio track. And I will say something that will probably surprise most of you: up until age 10-12 I never noticed the lips didn't match precisely the voices and I never even realized it was dubbed(!)

Even when I learned most of what I was watching was dubbed (there is so much American content on TV) it did not seem obvious to me. It certainly never bothered me. Maybe my brain subconsciously realized watching the lips was not giving out much information, so I learned to ignore lips.

Even today as an adult, if I don't know the actors and I watch a French dub, I am not able to instantly tell it is a dub. I have to watch carefully for at least 10-20 seconds to make a guess. Maybe this speaks more of the fact that French dubs are actually really well made (they will choose translations so that French word cadence matches the English word cadence). On the other hand I have watched other dubs, especially non-English movies dubbed in English, where it's blatant that they were dubbed and it is very distracting.

In any case, I always get a kick when I tell my American friends that I watched US content dubbed all my childhood and it never bothered me :-)

German dubs are smilarly well done, but the result often sounds slightly like the original language in tone and cadence, which is recognizable if you know what to look for. Someone I know can do a quite funny exaggerated version of German-from-English.

Extreme example: children say "maMA" (strongly stressed on the last syllable, opposite of German) in dubbed French movies. It's actually a little annoying, haha.

Interesting, it would be interesting to see if people who have grown up with dubbed movies pay less attention to lips when listening.

I'm french and I'm terrible at understanding speech when in a noisy environment for example. But I didn't grow up with tv so that's probably not it.

I would love a way to regularly discover and watch some top, 'hit' TV shows or movies from outside of America or Australia (my home country), or sometimes England. I have no issues watching things with subtitles, but it is quite difficult to organically find things to watch, that perhaps have 'not English' as their primary language.

For example, I will now look up the UFO Sweden show to see if I might like to watch it -- because I discovered it through here.

I have tried subscribing to a few different regular streaming services, but none seemed to work even if I pointed them in the right direction. I'm really not particular about which country of origin the production has, so long as it's "the best" or "very popular" in recent times from that place, it's worth me checking the genre and style to see if it piques my interest.

Any advice on this? [ nb. I have a similar issue with podcasts! ]

You might enjoy https://easterneuropeanmovies.com/, https://sovietmoviesonline.com/ or https://asian-movies-online.com/. They have both movies that are popular hits and that have been important to the craft and history of the art. Subtitles are usually from open sub-style sources so not professional quality, but I find it sometimes help with learning the language when it's more literally translated than idiomatically.

Sovietmoviesonline, eh?

Got any soviet movie recommendations that isnt straight up propaganda material? Did they even produce non propaganda?

What's wrong with propaganda? Is that a reason why you haven't seen for example Apocalypse Now?

You might want to watch Andrei Rublev, at its inception a controversial movie heavily disliked by the Soviet state:

https://sovietmoviesonline.com/drama/andrey-rublev

Edit: And then the post-soviet Fairy, which ruminates on the legacy of Tarkovsky's masterpiece.

https://sovietmoviesonline.com/fantastic/fairy

The rest of Melikyan's movies are also quite interesting and fun.

They also distribute post-Soviet russian movies and television series. You should watch the two takes on The Master and Margarita, both high profile projects based off one of the most beloved books in Russia, from two very different times in russian society:

https://sovietmoviesonline.com/drama/master-i-margarita-mini...

https://sovietmoviesonline.com/melodrama/the-master-and-marg...

The Soviet was especially good a adventures for kids and science-fiction, of which you'll find a lot there, for example Alice in Wonderland and Hard to be a God. You should watch the takes on the latter to get a feel for differences between soviet and post-soviet societies. If the younger version feels hard to watch you might need to watch The Green Elephant and other movies from the russian nineties to prepare yourself and better understand the feelings and mindset of the time.

And if you want you can watch old propaganda movies about the Donbass or whatever, you'll learn a lot about places you likely know very little about.

> What's wrong with propaganda?

Really? What makes you even type that out.

Movies done behind iron curtain, despite (and sometimes thanks to) everpresent censorship and oversight ended up with pretty deep existential messages that avoided usually not that brilliant censors.

Bear in mind that most artists abhored oppression even in toughest times, they are all free spirits after all, and I can imagine also in current russia there must be at least large pockets of similar folks.

Of course there were more shallow mainstream things more or less full of propaganda, and many actors overlapped between 2 categories. If one was clearly in critical camp then no work in industry, either some menial soul breaking job, prison or kicking out

I'm learning a language, and I'd love to be able to filter Netflix titles by language, but they absolutely refuse to do it (I assume because allowing users to filter content would expose how bare the offerings are).

Netflix, and other streaming is absolutely the worst at stuff like this.

Another thing is Subtitles in every language. They ONLY show subs in a handful of languages, even though the subs are in every language. For instance, I live in Norway, it's common to see subs in only Norwegian, which makes sense. But imagine you're a tourist/immigrant, etc. Maybe you want to watch this Japanese movie, well, now the audio is in Japanese, and the Subs are in Norwegian.

The app knows your language preference... but shows the subs for the country you're in. Even though Netflix/etc has the rights to the streaming.

It's baffling to me, and I must be missing a piece of the puzzle. But I'm forced as someone who's still learning Norwegian, to skip all Non-English media, OR pirate them so that I can get subs in whatever language I choose. (Yet another example of how piracy is the better UX)

I am really curious how it handles the Norrköping dialect in this film!

In particular the SMHI security guards "stället måste outoymmas, det finns en bomb i huset".

As someone who doesn't speak Swedish, what makes this particular passage interesting, and what is it about the Norrköping dialect makes it interesting?

(I have attempted to study Swedish briefly, but I feel like I need to do more sustain discovery of content to consume in order to keep me engaged long term)

I'm going to make a guess.

I suspect the question is, will the translation to English convey the regional dialectal nuances of the original Swedish?

Decades ago I read one of Douglas Hofstadter's writings about Gorbachev's accent. (As least, I think it was Hofstader. He wrote a lot about issues of what it means to translate.)

Gorbachev had a distinct southern Russian accent, which affected how Russians viewed him.

"Gorbachev’s southern speech is held against him." - https://time.com/archive/6732598/mikhail-gorbachev-3/

More strongly, one Redditor writes "Gorbachev couldn't fucking pronounce Azerbaijan for a living, he sounded like a self-important moralistic buffoon." - https://www.reddit.com/r/AskARussian/comments/tw3zlm/what_do...

However, Americans mostly heard Gorbachev through translation.

Should the translator use General American English? Or use a US Southern accent? Perhaps a Foghorn Leghorn accent?

When you watch a Russian movie in English, do you expect English with a Russian accent (often fake), like in The Hunt For Red October? Or do you expect to hear people talking with strong regional US accents, so characters from Saint Petersburg might have, I dunno, a Chicago accent?

We see this in Peter Jackson's movie interpretation of "The Lord of the Rings", where the actors used regional English accents to portray the social and economic standing of characters who spoke Westron.

Based on what I read about this AI-assisted translation, they are using the original actors but speaking English, which means the dialectal nuances will not be interpretable by non-Swedish viewers.

Models need a large set of training data to work properly and dialects are usually underrepresented in the training data.

The National Library and public service television has done some work in this area.

You can find models here:

https://huggingface.co/KBLab

More information in swedish about the process here:

https://www.kb.se/samverkan-och-utveckling/nytt-fran-kb/nyhe...

Its not handling the dialect at all? Its just 'painting' over mouth movements in sync with the english voice dub.

I support this simply just on the fact that maybe they wouldn't have remade Danish film "Speak no evil" (2022) which was such a great movie into one of the worst movies I've seen this year, the American film "Speak no evil" (2024)

Do I have to be the old guy that brings up Clutch Cargo?

https://youtu.be/J8P3vZo-0Hc?t=169

I feel like Clutch Cargo was only the most successful (however limited you define that) of range of cartoons that used that technique.

The very few shots they showed in the demo video, which one could assume were the best ones they wanted to highlight, do not look good, like it's not as bad as Ice Ice Matrix [1] but it does not look good.

1. https://www.youtube.com/watch?v=gnEIeVWLtbU&pp=ygUOaWNlIGljZ...

It looks pretty bad

https://youtu.be/PTngv5MmtXo?si=cCQcYtRaja5pPP6a

???

https://youtu.be/PTngv5MmtXo?t=105

They're admittedly pretty cagey with the parts they're willing to show, but the parts they did show look fine to me. Are you referring to the parts when they showed the 3d mask overlay on top of the characters?

I agree with the parent poster. They've chosen the best examples to show off, and yet all of them looked "off" to me. Perhaps because I was paying close attention I was more likely to notice it, but there was definitely an uncanny valley effect to it all.

To be fair a lot of the same applies to conventionally dubbed films, and then some.

It kinda reminds me of some of the old multiplayer PlayStation games that would flap your characters gums when you used the mic.

The motion lines up with the words more or less but it lacks any sort of emotion or emphasis like you'd see with real actors

It's very difficult to judge from these tiny clips. They're supposed to be showing off the technology with this video, but choose many clips with very little dialogue.

A few examples are very "gum-flappy"

A 3 minute previewing the technology that shows actual ~4s of "dubbed" footage.

Seems legit lol, nothing to brush under the rug definitely.

Man, this is so much better than many Bollywood movies that are re-dubbed. When the actors go back to the sound studio to re-record their lines, they don't match up with their lips exactly, and it can be off-putting.

Why wasn't the interview done in Swedish and processed with this technology?

Because "this technology" only changes the lips visually, and does not do any translation or dubbing.

> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.

Reminds me of the minor controversy about The Brutalist using AI to fix up Adrien Brody's Hungarian speech. Done by respeecher.com

https://www.vanityfair.com/hollywood/story/the-brutalists-ai...

A bit off-topic maybe: I really think the earlier film made by the same production team (Crazy Pictures), "The Unthinkable" [1], is a much better film than "Watch the Skies".

But apparently IMDb does not agree: "The Unthinkable", IMDb rating 6.0 "Watch the Skies", IMDb rating 6.4

[1] https://www.imdb.com/title/tt5227746

It's an algorithm. Why are we calling it "AI?"

Did they literally ask a machine "visually dub subtitles into this movie file for me please" and get a result? No? They didn't use AI then. They used a purpose built model in a purpose built tool to perform a single algorithmic function. how that algorithm works or what technologies it's similar to are not relevant.

/curmudgeon

We've had things we call "AI" since the 1950s. We only had text prompting as an interaction mode since ChatGPT came out in 2022. Why is prompting your test for whether something is "AI" or not?

> We've had things we call "AI" since the 1950s. We only had text prompting as an interaction mode since ChatGPT came out in 2022.

This is not even close to true. SHRDLU would be one prominent example.

> Why is prompting your test for whether something is "AI" or not?

What does the initialism AI stand for?

TIL about SHRDLU. But regardless, when you say "AI" in 2025, it means something broader than that. The root of the tree may look like trying to pass the Turing test, but many of the leaves don't.

Call it what you want, but there are lots of video processing tools out now with a technology lineage which includes generative/FM/LLM/NN DNA. Not sure about this one, but I know for sure of others.

> It's an algorithm. Why are we calling it "AI?"

And inference isnt? Explain your reasoning pls.

AI is just a marketing term for software at this point. It's become essentially meaningless.

As someone that grew up in a Latin America country watching dubbed movies my whole life I welcome this technology. Dubbed movies are usually terrible with subpar performances and recycled voices.

It sucks it will kill a large field for actors, but AI has the capacity to be much better with voices and performances that mimic the actors.

Note that for this movie, the actors are recording the translated lines manually. The AI is just there to change the lips.

I think this is gaining a lot of awareness, for example Derbauer (English channel) used Visual AI translation here: https://www.youtube.com/watch?v=UlZWiLc0p80

I found Swedish quite easy to understand, eg in this classic: https://www.youtube.com/watch?v=Fw4ylHuBOVk&pp=0gcJCdgAo7VqN...

> AMC Theatres will screen a Swedish movie 'visually dubbed' with the help of AI

So, a new version of the Python's "Hungarian phrasebook" all over again.

ugh, I don’t think it’s a bother to watch unsynched footage with a dub or just read the subtitles the whole movie through

What WOULD bother me is AI artifacts in a movie I paid to see in theaters.

So, only the video was changed to match real dubbed audio? Or synthetic audio?

Only a matter of time, for better or worse, until translation, synthesis, and visual modification all done with AI.

TFA:

> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.

the article has the answer

I support this, if only because I prefer dubs over subs, and this will make dubs much more worthwhile for the studio to get right, as well as better viewer experience.

I can't wait for the corridor crew episode about this

Its a cool movie but boy I wish people just learnt to read subtitles.

Firefox Can’t Open This Page

To protect your security, guce.engadget.com will not allow Firefox to display the page if another site has embedded it. To see this page, you need to open it in a new window.

Learn more…

Works for FF 36.0.1 on macOS.

Works for 136.0.2 on Windows

It's a pretty good movie.

Anything but subtitles lol

s/AI/work stolen from artists, actors, and writers/

They had the original actors record the lines to dub. They didn't necessarily have the budget to re-shoot the entire movie with them saying the lines in English, so that's not work lost. The "AI" here is a program that manipulates their faces to match the English syllables. All I see here is the creation of more work for people to do, not less (dubbing/subtitling it would likely bring a smaller audience).

Reply on news.ycombinator.com