Hacker News Reader

Comment by sorenjan | Hacker News Reader

I know Germany dub most video, but wouldn't a seven year old be able to read subtitles? It's a great way for her to learn English, it's how most Swedes learn it before starting school. I think there's a pretty strong correlation between countries' average English proficiency and how common dubbing is.

https://haonowshaokao.com/2013/05/18/does-dubbing-tv-harm-la...

Edit: I forgot to mention that the samples on the website is impressive and well made. How do you do the speaker diarization and voice cloning?

View on HN

Yeah, this isn't really helpful for her to learn English. This is more when we watch The Anatomy Lab, or BBC's "The Incredible Human Journey". She'll already be asking me a lot of questions about the content. So if I had to translate on top of that, it would be tedious.

Subtitles - those are actually being generated as well. I've generated SRT files during development. Color coded by speaker, and on a per-word basis, for me to get the timing right.

Basically, if you have a YouTube channel, you can take any video from your channel, run it through Speakz.ai, and you'll get 15+ additional audio tracks in different languages, plus 15+ subtitle files (SRT).

Voice cloning and speaker diarization was a bit of a challenge. On the one hand, I want to duplicate the voice that is being spoken right now. On the other hand, sometimes "right now" is just a short "Yeah" (like in the Elon interview) which doesn't give you a lot of "meat" to work with in terms of embedding the voice's characteristics.

Right now, I'm using a mix of signals:

- Is the utterance padded by pauses before/after? - Is the utterance a complete sentence? - Does the voice of the utterance sound significantly different from the voice of the previous utterance?

It's a deep, deep rabbit hole. I was tempted to go much deeper. But I thought I better check if anybody besides myself actually cares before I do that... :)

I think it’s a cultural difference. I’m also from a non-dubbing country (Netherlands) and I can’t stand dubbed content either. On the other hand people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it.

"people tell me they can’t stand subtitles because it “reveals” what they’re going to say before they say it."

I love watching movies in the original language, but this is something I hate as well, but something that can be avoided.

Some movies get it right, though. The timing, just the words that are spoken and even different colors for different persons speaking (very rare, cannot even remember where I have seen it). That should be standard, but with most movies you can be lucky if the subs even match the plot and do not reveal too much.

>different colors for different persons speaking

BBC iPlayer does this for some content, I don't know if it's ever on movies though.

It is. The iPlayer subtitles for Citizen Kane use colour to distinguish speakers.

Some of the best subtitles I've ever seen were on Tom Scott's YouTube channel. They use different colours, indicators for jokes and sarcasm, while also staying relatively close to what's actually been said. They're better than many big-budget movies and TV shows I've seen.

He talked about subtitling at some point, and I was surprised how cheap subtitling services are. I think he went beyond the price he mentioned, but it really made me question why big, profitable YouTube channels aren't spending the small change to do at least native language subtitles that Google can translate, instead of relying on YouTube's terrible algorithm

That said, Whisper seems to generate quite good subtitles that take short pauses for timing into account, but they're obviously neve going to be as good as a human that actually understands the context of what's being said.

Whisper can also generate timings at the word level, which you could use to make better-timed subtitles

Yes. But Whisper's word-level timings are actually quite inaccurate out of the box. There are some Python libraries that mitigate that. I tested several of them. whisper-timestamped seems to be the best one. [0]

[0] https://github.com/linto-ai/whisper-timestamped

That's a great use case for LLMs, actually. Translate the sentence only up to what has been said so far. Basically, a balance between translating word-for-word (perfect timing, but terrible grammar) and translating the whole sentence and/or thought (perfect grammar and meaning, but potentially terrible timing).

With the SRT file format for subtitles, I think, there's no reason why one couldn't make groups of words appear as they are spoken.

Actually, I have to do the same thing when generating the dubbed voices. Otherwise it feels as though the AI voice is saying something different than the person in the video, especially when the AI finishes speaking and you still hear some of the last words from the original speaker.

Unfortunately not all languages follow the same sentence structure, so translating "up to what has been said so far" is not possible.

Assume 2 dramatic stops in an English sentence, and observe Turkish version. You can "I will.. go to.... the cinema" "Ben... sinemaya... gidecegim" (I .. to the cinema.. go)

I am sure there are smarter examples.

I prefer subs over dubbing for foreign languages, but I cannot stand closed captions (for people who can’t hear at all) because having your eye drawn to the bottom of the screen for a description of something I don’t need to know about is horrible!

Sometimes it's hilarious when they're trying to describe the dramatic tension from sounds or music, and "reveal" all the cliches, though. "Music swells to a tear-jerking crescendo"

I think you get used to it. Like a punchline I've read, but I don't "register" it until the proper thing happens on the screen.

I'm Norwegian, and Norway used to be near-universally non-dubbing other than for TV for the very youngest children, and even then almost exclusively cartoons or stop motion etc. where it wasn't so jarring. But the target age of material being dubbed has crept up as it has become relatively-speaking cheaper to do compared to revenues generated in what is a tiny market.

The thing that annoys me the most about it is that it often alters the feel of the material. E.g. I watched Valiant (2005) with my son in Norwegian first, because he got it on DVD from his grandparents. He doesn't understand much Norwegian, but when he first got the DVD he was so little that it didn't matter. A few years later we watched the English language version.

It comes across as much darker in the English version. The voice acting is much more somber than the relatively cheerful way the Norwegian dub was done, and it while it's still a comedy, in comparison it feels like the Norwegian version obscures a lot of the tension, and it makes it feel almost like a different movie.

I guess that could go both ways, but it does often feel like the people dubbing something are likely to have less time and opportunity to get direction on how to play the part, and you can often hear the consequences.

A 7yo can barely keep up with subtitles in their mother tongue, depending on the speed. And that's probably true for a p90 reader. A p50 there is no way it can follow subs understanding what they say. Now, being a video, they might be able to interpolate from what they see, so it might be a nice challenge. But doing this with subtitles in a foreign language is only for a few, privileged minds.

Source: father of a 8yo with VERY good reading skills (already reading books in 2 languages targeted at tweens)

Dubbing in Germany is horrible and pervasive. Even in the news and interviews. Subtitles are cheaper and better.

As others have said, it is better to expose kids (that can read) to the original language plus subtitles.

So in other words, your solution while technically great is pedagogical not wise. A typical geek approach to a problem ;)

The worst thing about dubbing is that it's more important for the translations to have roughly the same length and correspondence to the original mouth movements than to be accurate. So the original meaning is often altered, and you don't even know it because of course you have no easy access to the original most of the time. But unfortunately Germans are so used to dubbing that subtitles don't really stand a chance. There are a few cinemas here and there that show original-language movies with subtitles, and on TV there was one experiment that I'm aware of a few years ago (on Pro Sieben Maxx) to show TV series with subtitles, but it was cancelled after some time. AFAIK it's also more expensive to secure the rights to show English-language content compared to dubbed content.

> but wouldn't a seven year old be able to read subtitles?

No, they wouldn't.

I don't believe that most swedes learn English by reading subtitles before starting school.

> I think there's a pretty strong correlation between countries' average English proficiency and how common dubbing is.

That I agree with.

Most people in Belgium learn English through that before school.

Why wouldn't swedes?

You are saying that _most_ kids in Belgium can read subtitles before they start school?

It took me several years of school before being able to read fast enough to follow along subtitles, and the same for everyone I know.

They probably meant before they start learning English in primary school, not before they start school.

This used to be the case in the Netherlands too; I picked up a significant body of English from British TV series watched with subtitles as kid. Nowadays this advantage will probably be missed by most children, because the streaming services offer a lot of dubbed content, and you get to pick what you watch unless someone guides you. Subtitles can be avoided for longer.

I think you're native English and have the associated bias concerning how it works in practice?

Since kids first learn their native language ( write, read and speak) in school and only years after then ( mostly), learn foreign languages.

When they learned to do it in their native language, they hear English spoken on tv with eg. Dutch subtitles and pick it up. Sometimes before they have English lessons.

Most kids, as such, know a fair amount of English before they have it ( = English) in school.

The Dutch subtitles isn't always a requirement though. Kids will pick it up in some shows, eg. Pokemon would be a good example if English spoken.

Are you saying that kids of age six can understand and speak English at a basic level - say half way to A1?

Or is it just a basic familiarity (like a couple of most common words) and awareness that English exists?

EDIT: I see from a reply below by Freak_NL that it probably means before the kids start learning English at school. That makes more sense, as they would be older at that point.

> as they would be older at that point.

I don't know about The Netherlands but here in Norway children start learning English as soon as they start school at the age of five or six. But quite likely many of them will have at least some English already because of English language television, computer games, etc.

> I don't believe that most swedes learn English by reading subtitles before starting school.

It's not about learning the language per se, it's about familiarizing yourself with the sound of the language, which then makes formal learning feel much more intuitive. English becomes an easy subject because you always feel a little ahead of the material. When faced with a "fill in the blank" type of questions, you're able to answer them by what feels right, even when you can't quite explain why it feels right.

It's why #1 rule of language learning at any stage in life is always gonna be immersing yourself with the language you want to learn, and by far the most effective way to immerse yourself (excluding moving to another country) is to consume content in your target language.

>> but wouldn't a seven year old be able to read subtitles?

> No, they wouldn't.

hard disagree

Reading speed at that age will vary greatly. Reading subtitles while also having to follow the picture takes away focus and that makes it hard much harder for an inexperienced reader. My daughter, who picked up reading very naturally would have been able to follow sub-titles at age 7 without much trouble. My younger, 7-yo son on the other hand, who is more average in reading ability wouldn't be able to keep up with subtitles yet. Average reading speeds at age 7 seem to be 60-100 words per minute where subtitles are more at the 100-150 words per minute range. So for above-average readers, it will be possible but for the average, they won't be able to keep up consistently.

My 6 year old has been watching 20 minutes of cartoons every night for the past two years. This is the only exposure to the English language that she has ever had.

She has learned to understand what is said in the cartoons. Of course she misses some things, but it's surprising how much she gets.

Like, when I ask her "what did Bluey just say?", she can explain it.

Children's brains are awesome.

But actually, grown-ups can also pick up quite a lot if they actually immerse themselves.

Bluey is an excellent cartoon to do that with. Kudos!

I just wish there was a way to buy the Australian original version as a download.

Subtitles in a foreign language? Probably not. Subtitles translated into their original language? I think it's probably an exaggeration that people have learnt it before starting school because it implies a lot about what learning it means, but picking up a number of words, sure.

Young kids don't even need subtitles, their brains are wired to figure out spoken languages, after all that's how we all learn our mother tongue initially. Last summer my then 3.5 years old, to my huge surprise, started talking in (simple, but correct) English with some tourist kids she met in the park. We never spoke English in home with her before, so I presume she picked it up from youtube and her older brother, but I had no idea she can form full sentences - including conditionals and past tense. And at first she was a bit slow to express her self, but after a few hours of play with those kids she sounded totally relaxed and fluent.

>I know Germany dub most video, but wouldn't a seven year old be able to read subtitles?

I gotta say... while sometimes it is a necessarily evil, I would so rather not have to read subtitles. I often want to listen to a show so that I can also continue working on catching up on email, etc, IE: I can't read two things at once, but I can listen to one thing and continue working on something else.

Reply on news.ycombinator.com