AI could end foreign-language subtitles
axios.com
axios.com
Adding text to movies has a huge array of advantages: it preserves the original performances, it makes the dialog accessible to people with hearing or auditory processing disabilities, they can be multiplexed in a way that audio cannot, etc. Obviously it would be wonderful to have an 'autodub' option! But the times where one would use it seem quite narrow.
I believe that reading the Swedish subtitles and hearing the English words simultaneously have played a large role in helping Swedes to learn English.
However, it does help a lot that the grammar of Swedish and English is relatively similar. I have been trying to use watching anime to get better at Japanese, and the different word-order and clause-order in the spoken Japanese to the subtitles have often tripped me up — especially when longer sentences have got broken up into multiple subtitles.
That was the norm as I was growing up in Israel, with the exception being children's programming (usually animation) that was dubbed. The dubbing was usually quite good, and done by actors prominent in children's programming.
OTOH, being bilingual myself, the poor translation of subtitles was constantly driving me up the wall. I just couldn't ignore the mismatch between the dialog in English and the subtitles in Hebrew, but the rest of my family wouldn't agree to let me cover the bottom fifth of the screen.
Semantically, the translations were generally accurate except in infrequent cases where the translator obviously misheard something or translated a phrase overly literally (more often with non-RP British or Australian English than American), but stylistically, the translation was always bland and "correct" (eg. "sure" and "yeah" and "you bet" would all inevitably be translated as "yes").
I have tested out services that translate your "own voice" to another language and it was pretty cool, although I couldn't tell if the translation was quality. Automated translation isn't a solved problem, and this seems more of an issue than the tone of voice.
Now silent films aren't a problem, as the text is shown before / after a given scene. So there is no processing two things at the same time. And it isn't a reading problem in general, because I'm an avid book reader and can consume books quite fast (although I only enjoy them if I read them slower and contemplate each scene).
I can only say that I absolutely prefer subtitles 100% of the time because the original performance is usually better. Good voice actors are not cheap, besides the fact that the sounds don't match the lips when it's a different language speaking.
I figure most people could probably do it with enough practice, though.
But I can't imagine watching, say, a foreign live-action sketch comedy show subtitled. I'd have exactly the bad experience you're describing.
If anything, they are usually at the other end of the spectrum, in that most local releases don't even include japanese subtitles (To be frank, I'm quite surprised this is even legal, considering the impact on deaf people)
So more than for artistic reasons, I think it's more often in order to lower the number of quality key frames required, so budget reasons.
I also can think of several shows that achieved significant fanbase in contexts where keeping an eye off the subtitles can be hard, while still being appreciated for their visuals: Tatami Galaxy, Monogatari Series...
"Screen-original" films and TV series are written in more a "top-down" style, with scenes being planned for their pacing, and the dialog in a scene liable to get cut down or rewritten to keep said pacing.
"Adapted from a non-visual source medium" films and TV series also almost always get their scenes cut down, just by necessity (a novel won't fit in 1.5hrs.) Though only "almost", because sometimes a director sets out to adapt a work into a really long treatment that can fit all the source material and then some — like the Lord of the Rings movies, or Game of Thrones. But even then, the director is still someone who is immersed in the top-down, "we get to knit it together how we want" zeitgeist of regular Hollywood film&TV production.
"Adapted from a visual source medium" films and TV series, though, are — if not heavily budgeted for adaptation polish — written mostly "bottom-up", starting with a treatment and storyboard that keep every line and shot from the source, and then letting pacing slip to keep in all the stuff that existing fans of the source material expect to see (and because the producer wants as many episodes as possible, so why not stretch one chapter out into five episodes if you can?)
This can lead to some extremely long shots, where there was dialog in a comic-book panel that was clearly not originally written with the intent of having it performed while holding/panning on the exact panel shown; but where a lazy "reading" of the source by the cinematographer leads to exactly that.
You can also call this the "make a radio play / audio drama out of the book; lock it down; and then add visuals to it" adaptation strategy.
(This is also common, as it happens, to specific genres that started out in the radio era. True crime, whether it be on TV or on YouTube, mostly consists of podcasts with the visuals almost being an afterthought. This is why so many people put it on in the background: the way it's done, you don't have to look at it to follow what's going on.)
The only downside is you have to actually _watch_ the show rather that staring at your phone most of the time...
You have multiple brothers with one girlfriend? I understand this is not likely, but it's such an interesting concept I kind of want it to be true.
I mean there's some undeniable fact that the original text is the original thing, and low-quality translations often just remove entire parts of sentences or details just because it's "annoying", and you are missing out on lots of fun little details. Not super relevant in paint-by-numbers TV shows but for shows with a lot of flourish in the dialog...
However, moving pictures will always be an artform that are heavily reliant upon the picture part. With subtitles, I lose my ability to focus on minor but important details in film. During dialogue, this includes subtle facial expressions. I also pay less attention to all other visuals, which I find difficult to do without subtitles off entirely. I feel like I miss out on a lot of potential beauty in cinematography with subtitles enabled. (Although, to be fair, there is usually less dialogue in parts that emphasize on cinematography / setting.)
Anyway, just throwing in my two cents. I used to think people who disliked subtitles were expressing some broader dislike of foreign films, so I found their opinion perplexing or even stupid. But I've come to that side of the aisle myself, not because I dislike foreign films, but because I realize that reading a small bar of text at the bottom of the screen will always handicap my ability to take in the original experience as intended by the director of the film (in 99.9% of cases).
Heck, they're not even limited to foreign languages. I put them on in English because I'm starting to lose my hearing now in my late 40's and if I turn up the volume loud enough that I can hear the dialogue ok, the sound effects would be so loud they'd set off the neighbor's car alarm.
They are an absolute boon for those interested in learning foreign languages.
- family is sleeping so you keep volume low
- hard to understand accents or lingo
- bad audio mixing or bad audio systems that muffle words and rest of the sound
It's easy to tell which transcription services have access to the original script as a reference and which don't. I have subs on most of the time to catch subtle or barely-audible dialogue (or to decipher accents). There's sometimes when I can tell "oh gee that's not right at all but I can see how you misheard it." And there's other times when dialogue is coming from offscreen or a TV/radio and it's too far in the background to understand unless you are reading along with the script.
It’s actually funny seeing how frustrated with subtitles my American girlfriend gets when we watch anything foreign. “You mean this basic skill every 7 year old has is hard for you, an educated super smart 30 year old? Bwahahaha”
I still prefer watching everything with subtitles. Actors mumble a lot.
With subtitles my eyes are drawn more often to a part of the screen where nothing visual is happening. They aren't glued there, but it does create some bias. I have occasionally rewatched a scene without subtitles and noticed a lot of subtlety in the performance I had missed the first time though.
That said, a bad dub is at least as distracting. Probably much more so, if anything. I would rather watch a bad sub than a bad dub, though I have occasionally enjoyed dubbed things as well. I don't think it's an all or nothing thing, but "never watch dubs" is the hard line heuristic that makes sense to me if you're gonna go that way.
Crucially, outside of preference, subtitles are occasionally necessary for translating text on the screen (e.g. on building signs, newspaper front pages, posters, chalkboard writing), as information is conveyed solely through writing without spoken dialogue.
For example, consider the chalkboard scene in "Komi san can't communicate": https://www.youtube.com/watch?v=JoH8I-NC5tM . It's key to the story that a character has trouble communicating verbally, so subtitles are essential to understand the written communication on the chalkboard. (You can then compare to a more extensively subtitled version that translates all the text on the board; good subtitles make a difference to storytelling: https://old.reddit.com/r/anime/comments/qdjgba/the_blackboar...).
Yep I have no problems hearing but newer movies and shows have uneven mixing where it can get really hard to hear over the background noise and subtitles come in clutch constantly for that
I don't like dubbing because it's generally much lower quality than the original performance, but I get why people who don't have a problem reading could prefer it.
French have great dubs and most of the movie released are watched dubbed.
Maybe it would be nice to have subtitles that appear a word at a time as they are spoken, but I imagine that's not very practical for various reasons.
https://youtu.be/R3shsgKbd_M?t=12s
That's a pure text format; no bitmaps involved, just markup and TTF fonts. It can do vector shapes and perspective positioning. These are somewhat overdone examples, but one more common nice use case is literally translating signs on buildings, objects and such, seamlessly, while preserving the image style. The whole thing is rendered in real time and you can toggle the subs off and see the original frame.
Again, that's all markup rendered on the video in real time from fonts. Yes, really.
It is the standard in the anime fansub community, but has seen little adoption elsewhere. Partially this is because it seems commercial operations rarely have much interest in subtitle quality or technology.
Incidentally, even YouTube supports an undocumented subtitle format that can support a decent amount of styling (YTT). Unfortunately, support is inconsistent across web/mobile versions, and it looks best when viewed in a browser. I've used it to do things like this (turn on captions):
https://www.youtube.com/watch?v=dh1BCyjCbes
https://www.youtube.com/watch?v=IxLI7KmzOjE
You can convert a subset of ASS to that format with this tool: https://github.com/arcusmaximus/YTSubConverter
The other day I happened to see live content with closed captions, and I thought I would love the option to see subtitles like this, where words are translated precisely when spoken.
You generally only see word for word matches with subtitles when the subtitles are in the same language as the original lines spoken by the actors.
Since I’m moderately familiar with the other language, I really just need the translation for words I don’t understand or mis-hear.
There is such a thing as a real-time translation. I’ve heard it in news broadcasts when a politician speaks a foreign language. It also happens all the time at the UN and other international negotiations. It works well but has downsides such as occasional latency while the translator needs to hear a few more words before they can translate.
Something that might be sort of like what you are looking for is Language Learning with Netflix https://chrome.google.com/webstore/detail/language-learning-...
It's a browser plugin that provides advanced subtitle functionality. You can have dusl sets of subtitles up, or just the original language subtitles with the option to pause and mouse over individual words for the definition.
Different languages have very different grammar, and even the best translated subtitles won't be able to handle a clever usage of the original language's grammar.
This is one of the many tradeoffs you make with translated works. You always lose something (sometimes a lot) in the translation.
I think there's a lot of gatekeeping about this. People - perhaps rightfully so - believe subtitled versions to be more ... "pure" than dubbed versions.
I don't watch any language dubbed movies, but I wouldn't chide anyone who did. Not everyone wants to spend the whole two hours glancing up and down across the screen, nor are they necessarily wanting to learn a language passively, etc.
So I think there's absolutely a perceived problem, it just doesn't apply to all viewers.
Well, they are more pure than dubbed versions. There is so much nuance and tone in the original language that is either left out or changed in the dub. Not to mention word order (subject-verb-object in English/Chinese, subject-object-verb in Japanese/Korean) that further changes the timing of impactful words.
I am not looking forward to having to watch the dub versions of Totoro and other Miyazaki films when my daughter is between 2-7 years old...
Keeping the original audio track does preserve the direct relationship between the actor's tone and expressions, of course, and it could also be something closer to what the original makers intended even if the voice actors in a dub also do a good job.
The audio track also feels somehow more right and authentic when someone who's supposedly Japanese speaks Japanese, or when a supposedly French person speaks French, or when people in a show set in California speak English rather than somehow magically speaking your native language.
Then there's of course the question of dealing with scripts that have multiple languages spoken in them. Something would surely be lost if multiple languages were dubbed into the same third language. It's often easy to recognize the different languages being spoken even if one does not understand them.
Though this is much less of an issue when watching fansubbed media (for Japanese content, anyway) because the fans have translator notes and do a much better job than Netflix or media studio translators.
I've sometimes thought that languages could be indicated by phonetic rendering of an accent (an audio version of this was done on the excellent and clever show 'Allo 'Allo where dialog in 'French' is English with a French accent, dialog in English is RP, British characters badly speaking 'French' use RP with funny word substitutions, and so on), similar to how comics often indicate other languages in word balloons with conventions such as <angle brackets> or italicization, but that's probably too high a bar.
Famously, the closing dialog for Breathless, er, À bout de souffle doesn't translate to English.
https://www.youtube.com/watch?v=peS6ADDysms
https://en.wikipedia.org/wiki/Breathless_(1960_film)#Closing...
(It should go w/o saying reading this is a small spoiler and you should most definitely see this film first if you haven't. I happen to think the American remake is trash, but my dad loves both it and the original. In any case, it is among a small handful of films that I consider perfect.)
And for the record, Chinese also makes heavy use of SOV, eg 他們昨天到韓國去了。
I really enjoyed the cinematography of a film the other day, but the music completely ruined it.
it was an advantage growing up watching Hollywood movies in English with Arabic subtitles, it helped me learn the language.
I suspect it simply won't work - translating accurately is really hard, but translating elegantly is frequently impossible. Some things are just easier/punchier/funnier to say in language A than B
https://www.dailymail.co.uk/sciencetech/article-9613749/AI-c...
When the occasional rare case arises where they need to watch a foreign film/show, they find it straining to both read the subtitles and watch the film, and so they conclude that subtitles are bad or “not for them”. In practice this stems from lack of practice — the perceived conscious effort of having to read subtitles while watching a film disappears entirely after you have done it a few times with sufficient regularity, and becomes completely effortless, subconscious, and automatic. But this is a skill that needs to be developed. Saying that subtitles are bad because they distract from the film is like saying that riding a bicycle is bad because you fall all the time, when in reality you just haven’t learned to ride one.
So maybe I watch the kind of stuff where that kind of performace is required. Maybe it has certain genres or mediums where dubs work well like Romcoms or Marvel movies.
With subtitles one has to constantly be looking at the actual show itself. Furthermore if one makes use picture in picture such as on a mobile platform or a tablet device the subtitles are so correspondingly small as to be basically hieroglyphic at that point.
That said, other people can obviously have their own preferences. Just sounds so weird to me.
I have no doubt these solutions will be popular, because many people have difficulty reading and will prefer audio when given the option. I'm just confident they won't reach any measurable quality and the translations will be full of non-sensical/anti-thetical statements.
Also, languages are bridges, not barriers, by exposing yourself more to another language you will learn it if you don't reject it.
I have friends who religiously watch subtitled anime only and they are no closer to learning Japanese in any real sense of the word than a person who would watch the dubs, except for picking up a stray random word here or there, but knowing that kawaii means "cute" doesn't give you even a superficial understanding of Japanese. For that, you need to at least pick up hiragana and katakana.
Learning a language requires deliberate work. There is no shortcut.
I speak Spanish. Subtitles have helped me a lot when languages are similar. I watched three seasons of a Catalan TV show subtitled in Spanish, and now I can follow conversations and read it. Before that I wasn't able to understand spoken Catalan.
Some language pairs were this may work are:
- Spanish/Catalan
- Portuguese/Spanish
- Dutch/German
Um, that's how babies learn, I think there is plenty of evidence in every baby on the planet.
People stop learning this way when they start thinking of languages as barriers.
They (obviously) also work less well for slower readers.
Well yes because it's not exactly novel or complex. My point isn't that there is no rationale it's whether we should just blindly go along with it.
FTFY
Coming from Germany, this wasn't always the case, but nowadays watching dubbed content feels so weird and impoverished, I don't even want to imagine what it'd be like knowing that the voice is fully synthetic.
There is something about watching a movie/show in its original language that you cannot understand. I think its awesome to be immersed in a different world, getting the sounds, inflections, etc from a different language of yours. I will always prefer subtitles to dubbed versions if possible.
I understand some people don't, but I think it's just a matter of being exposed to it. And probably the younger, the better.
There should be a disclaimer on this advertorial piece.
Dubbing movies is common in big countries where the locals speak nothing else than their own language. Here in Germany, you have to pay attention if you go to the cinema but they do have original language movies as well. Lots of Germans go to see those instead of the badly dubbed versions. I tossed out my TV ages ago because I don't speak German very well and don't like watching dubbed content. Especially if that content is originally in English which I actually can understand.
One annoyance with the local netflix here is that while they have most content with the original sound track, they often swap out a lot of the text in these movies with German text. E.g. movie fragments showing a news paper or somebody's text messages, or the famous rolling text in star wars. And of course a lot of American movies or series sometimes feature people not speaking English. Netflix solves this by offering German sub titles. And sometimes those are actually baked into the movie.
Par of the issue is that there's a big dubbing industry that is incentivized to keep on dubbing everything whether people like it or not. AI will decimate how profitable that business is.
This is my pet peeve about streaming services. Why would they have only German subtitles in Germany? The licensing costs of all major languages can't be prohibitive. I only watch content that has English subtitles -- Netflix has thankfully improved on including them, but otherwise you're mostly out of luck. Even Blu-rays need to be imported if you want to have English subtitles on English movies!
It's not the cost, it's that their removal is a disincentive for people to use VPNs to hop outside of their usual residential territory.
I suspect there's some estimation going on to figure out what languages someone is most likely to know given a certain interface language and location. This is a somewhat common annoyance language learners run into because it's not obvious that you need to go through these hoops to get access to the language you'd like to use.
I also respect actors and actresses enough to appreciate their performance includes their original voices.
I'm a fast reader and subtitles are like a second nature to me, to the point I don't "remember" having read them (likewise I sometimes don't remember if I read a novel in English or Spanish unless a particular turn of phrase caught my attention).
About the only thing I agree with is that dubbing (barring some honorable exceptions) sucks. I did listen to a part of La Casa de Papel's English dub -- the horror!
In short: do not want this.
These days I have gotten a bit faster at reading and I got things like more readable fonts to help me (also, some subtitle systems are just terrible, I often torrent stuff that is on netflix despite having an account because their subtitle system sucks).
Yet, I am from the Netherlands, and here, with basically nothing except media for children being dubbed, I have gotten better at reading them and used to it. And honestly, despite the effort I have to go through, I totally agree. It is totally worth the extra effort. I'll always prefer the original language, no matter the quality of the dub.
Language is just an important part of the culture of the original content.
1. AI might be able to prevent bad dubbing/subbing, but it will not be able to compete with _good_ dubbing/subbing in our lifetimes. There's an artistry there that no AI can match.
2. I love subtitles and prefer original audio + English subtitles in anything foreign-language. Besides the voices matching up to the movement of the actors' lips and thus not breaking my immersion, I also adore hearing how familiar emotions are reflected in languages I do not know. It makes me feel more connected to humanity.
I'm partially deaf so I watch everything with subtitles or captions, and don't miss a thing.
This has been said a few times already and been proven false. The quality of GANs is moving at a staggering pace.
You can't base a metric on something with no measurement.
Show me the proof.
In Sweden, everything foreign is subtitled, and as a result of the vast quantity of subtitles produced, the quality is high, so I've grown up expecting good quality subtitles. And then I watch something on HBO or whatever, and the subtitles are just bad. Clearly not proofread, clearly done by someone who didn't watch the thing, and just translated the written material as is.
And the art side of subtitles is often neglected or plain just not understood. Timing is really important, which people understand when it comes to for example dialogue timing to make a joke land. But then the subtitles are shit, and ruin the joke because the subtitle timing is off, when you could have had good subtitles that worked with the material, preserving the timing!
Why aren't studios and networks investing as much in the subtitles as the rest of the production?
Yes, 95% of subtitles are easy to make, there's a single unambiguous way to translate what's being said into your target language, and by syncing it to the audio it's gonna be good enough. But some things are hard to translate, because there simply doesn't exist a 1:1 mapping between every language, despite Google assuming the world works like that.
But YouTube's auto-caption thing is garbage. It's enough to sort-of kinda make out what is being said, but it's often complete garble, and there's zero understanding of good timing or the art side of subtitling.
But somehow, subtitles are Not Art? It's somehow not creative work?
I'm a non-native English speaker, and I like watching English-language content with closed captioning in the original language.
There's often a lot lost in translation, especially on the screen (written content might have different issues), because it's often hard to get the rhythm and timing right without changing the contents in non-trivial ways when the source and target languages are completely different. Since I can understand English fairly fluently and nearly always get the nuance that would be lost in a translated version, I prefer the original language, even for the subtitles.
Since I'm not 100 percent, though, having the subtitles or closed captioning usually lowers my cognitive load. It also means I don't have to worry about missing something because of an unfamiliar accent, a very quickly spoken line, or some other random thing.
The default of you load it is dubbed, right? That’s what it started for me. So this stat probably has more to do with the default than anything else. They’re making a weak case that this problem even needs solving.
I don't think so, at least not for me.
- The hard part about subtitles is not translation it's contextualization. Languages are not variable replacements and words have different meanings based on the context of the sentence. Languages have nuance, phrases, and slang unique to their cultures which further adds difficulty.
- coupled with the above, permutations can get large with translated dubs. Which subtitle are you supposed to show? Most translated subs are done off the original audio track and the dubs are done separately. So for example with an English show, the English subs match the English track, but the translation will be off for the Spanish audio track. And the Spanish dubs will not match the Spanish subs either.
- lip sync on dubs is a hard problem for many reasons ( full face and body emotion, how words are said in the native tounge, etc..) and will significantly improve the viewing experience. You can do creative things like have fewer full face shots to help with this. It's also important to get the audio mix right so it doesn't sound like an extra track. Keep in mind you now need more videos to cache and store and the ability to swap video streams when you swap audio streams.
- animated content offers other approaches than leveraging nn/deepfakes because the assets are already rigged. The use of game engines offers some really interesting opportunities here.
That is, companies will try it, there will be some bad and ridiculous results, there will be pusheback and the tech will largely be ignored for a decade or so.
Main problems will be intonation and inflection. What is a question to an English speaker is a regular sentence to a Russian. And so on. "Petabytes of data from primarily English-speaking libraries" mean jack shit if you wnt to actually do voice translation into/from foreign languages
https://www.youtube.com/watch?v=ijwHj2HaOT0
This is live in production today - business owners are getting calls from robots, and you can trigger it yourself with Google Assistant.
[Disclaimer: I work(ed) on Google Assistant - I'm in the process of transferring out - and have listened to some presentations about how Duplex works. It's pretty crazy technology.]
Opera houses have subtitles on a separate text-only screen (or screens embedded in seats). Why can’t I have a device like that at home?
A narrow subtitle display that sits below the TV, sort of like the Touch Bar on the previous-generation MacBooks. I’d buy that.
You can install it on your phone and it'll subtitle your life if you can find a good place to put it down. It's meant for the deaf and hard of hearing (hence features like visual alarms) but in my experience it'll transcribe stuff that's being said pretty well as long as the mic catches the audio you want to hear. Perhaps it would be even better with a quality (bluetooth? 3.5mm?) external microphone.
Does anyone with more experience have any idea?
That being said, as I understand the article this marvel ai thing is supposed to replicate the original performance using the original voice in another language? Of course that would be fantastic a solution, but I doubt that the technology is anything near what it wants to do…
But one thing that bugs me, why is it that even today - we have to have programs with a hard-overlay of sign language interpreter in a you have it if you want it or not and the selection is limited. For those who depend upon such sign language, wouldn't it be great if they could watch anything and some AI would do the sign language for them. You would of thought that such a task would be easy, and the market is there but yet, it seems to elude us.
So my hope is that whilst this is useful, it may well open up avenues like the above in which these tools can be leveraged in more practical ways for a group of users who in effect have what they can and can not enjoy, somewhat overly limited as they are at the mercy and whims of the broadcasters who in many instances, only do them to quota tick some regulation put in place to at least not totally shut out a group of people who can't hear at all.
I find I don’t need the volume as loud and I get retain more of the dialogue.
Why can’t someone just loosely transcribe without time stamps and sync it to the video?
Most people read more slowly than they can listen. Many other people can read faster than they can listen. But you need subtitles that stick around long enough for the first group, and really, you want a comfortable margin so that people aren't frantically trying to finish reading every subtitle before it disappears.
I watched the show 大江大河 ("Like a Flowing River") on Viki, which has excellent community-generated subtitling. The second season isn't available on Viki - only YouTube. (Actually, Viki seems to have lost the license to the first season by now, too.) And the subtitles are abominably bad, bad enough to make me stop watching the show. But they're perfectly synched - every subtitle on Youtube is exactly matched, millisecond-to-millisecond, with a Chinese subtitle which it attempts to translate.[1]
[1] Ignoring the timing, which is much too fast for the English subtitles, the fact that each subtitle is translated independently is another huge problem. It leads to nonsense when one sentence is split across multiple subtitles, because the English and Chinese do not naturally present the same information in the same order.
All of the attempts I've seen of using audio information to synchronise subtitles have been awful. One issue is that some languages subtitle everything, even screams and incoherent shouts (such as Japanese) while others only subtitle dialogue and often rework dialogue for the purposes of making the subtitles short enough to be readable easily. It feels like you need too much domain knowledge to know how different languages subtitle things and that subtitles that match the general meaning of what is being said should be matched up.
I'm about to start experimenting with custom TTS generation, and I'd like to keep you in the loop if my efforts are successful.
We should invest in making people’s ear accustomed/used to hear other languages. It has great intercultural benefits.
In the case of English, the orthography is difficult, a large proportion of people are functionally illiterate, and even more have a tough enough time reading that subtitles just go by too quickly.
I had much better luck with Star Trek TOS, very little deviation between the English and German dialog, and the subtitles matched up well with the spoken dialog. Much better for learning - though “er ist tot Jim” doesn’t quite have the same punch IMO.
The only feasible outcome here is deepfaking the voice from the recordings from localized voice actors to the actual movie actors. Perhaps, in the future, a voice actor might be doing all of (1) translation (2) voice acting (3) deepfaking in one shot. One person might be able to dub the whole movie - alone.
But, for sure, we won’t be able to remove human from the process.
For example, I prefer to see subtitled movies in their original language (I'm a Spanish native speaker). But there are some dubs that are awesome, even better than the original: for example, Shrek. I'm concerned that eventually studios may just throw the script through Google translate and call it a day.
They are time stamped hence annotated precisely, hash and file matching makes media content to subtitle matching exact and overall there are language variations.
Timing is much more automatized nowadays, but still doesn't feel as high quality as handicrafted timing does.
Goupon founder Andrew Mason's new startup Descript does a fantastic job editing podcasts from text. This same sort of tooling is absolutely going to make it into film.
This sort of stuff is already trivially possible with zero effort:
https://fakeyou.com/w2l/result/WR:t5m9xayqcwkjf9wng3nb4sxgzh...
In ten years, we'll be able to reposition actors and fix flubbed takes. We won't have to visit set, use expensive lenses, or do many of the things that make Hollywood the domain of the rich and well funded.
In thirty years, we'll just tell the machines what we want to watch.
Why not just deep fake the entire movie without using any live actors or sets too?
It's a good idea but I'm not sure how well that would work.
https://knowyourmeme.com/memes/youtube-automatic-caption-fai...