Ask HN: Why don't podcasts/audiobooks have closed captions for misheard words? | Hacker News Reader

Ask HN: Why don't podcasts/audiobooks have closed captions for misheard words?

31 pointsamichail39 comments

This would be like automatically generated captions in YouTube but for audio only content.

Podcasts don’t do it because good transcription costs money. Audiobooks already do it, it’s called … the book.

And I’ve heard that 3rd party audiobook transcription violate’s the written book’s copyright (different publishers perhaps?)

It's not even related to the publisher, they're two different products that are separately licensed. Generally the publisher only retains some rights to the book and how they may use it and same goes for the audio book, with the difference that the recordings are owned by those that produced the recording and not the original writer (unless it's licensed as so).

Podcasts are taking over one off content like daily radio programs, so the quantity is huge.

Audio book subtitles could be more useful if they highlighted what was just read instead of the whole page. Possibly with TTS alternative pronunciations or spelling aloud upon request.

>automatically generated captions in YouTube

Youtube's AI-powered auto gen captions are often inaccurate for uncommon words and proper nouns (names, etc).

Here's an example to a deep link : https://youtu.be/z9IXfYHhKYI?t=20m33s

The instructor actually says "UIs" (user interfaces) but the auto gen transcript is "you eyes".

Therefore, accurate transcripts require humans to manually correct the errors and most content creators of podcasts don't have the time-or-money budget for that.

If your question is actually asking why the iPhone/Android software apps don't dynamically generate and display "close captions" while the podcast/audiobook is playing similar to Youtube or Chrome, it's because the devices don't include an onboard trained neural network that's big & comprehensive enough to translate general audio of uncommon terminology. Today's smartphones only have RAM of ~4GB to 8GB. (I read that iOS API has max RAM allocation of 5GB per app.)

I found a Google paper talking about reducing a 2GB voice recognition model to 450MB with alternative techniques but the neural net is probably not good enough for accurately transcribing uncommon words: https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...

So current state-of-technology means accurate close captions on smartphones would require a separate "subtitle track" (authored by humans) akin to *.srt files in mkv video containers -- instead of attempting dynamic speech recognition of the audio. Maybe not enough consumer demand to have mp3 or m4b files formats with transcripts embedded.

Live transcribe is available on latest android devices today https://support.google.com/accessibility/android/answer/9158...

iOS also appears to have 3rd party app for this

Why would mobile devices need to do this themselves?

Couldn't Google do this with a podcast/audiobook service that uses YouTube tech?

>Why would mobile devices need to do this themselves? Couldn't Google do this with a podcast/audiobook service

To answer your question, what's the baseline premise of how podcasts should work on a mobile phone?

- Should podcasts/audiobooks continue to be local files such as mp3/m4b that's copied to the mobile phone? If so, the companion technology to enable "captions" would also require a local transcript file that's copied to the phone. (Like srt files for mkv.) In that case, a Google podcast transcript service that created such text files would require the whole podcast ecosystem and app players to change and recognize timestamped transcript files. E.g. even if you embed transcript into mp3/m4b container files, Apple's Podcast player app isn't going to know about it. So would it require everyone to download 3rd-party podcast players? So this scenario -- while not impossible -- has a lot of barriers to adoption. Too few consumers probably care about it.

- Or should podcasts/audio be streamed from the cloud -- like Youtube videos -- to more easily enable your captions feature request? In this scenario, Google's podcast transcription wouldn't require embedding the text in new audio file formats for distribution. But you still have the mobile apps incompatibility issue of displaying the captions for podcasts. Again, Apple's Podcast app doesn't know about streamed captions. You also have content creators locked into Google's hypothetical podcast-with-captions-streaming cloud ecosystem which many wouldn't want to do.

Could be both right? Your podcast app could generate captions on the server (assuming, as mentioned above, that on-device hardware limitations prevent doing so on the phone itself) as a file that you download along with the mp3. They'd only have to generate one file for every episode of every podcast that anyone subscribes to, which doesn't seem like it'd take much hardware. There arent' THAT many.

Could even be one third party service they all use, like how all the audio players used to use CDDB for cover art, track names, etc.

Apple's podcast app is probably the biggest impediment to podcast technology, but no reason 3rd party apps couldn't all do this.

Other than economics. It would be a large expensive in terms of bandwidth, processing, and maintenance of the TTS service. It can be difficult to get users to pay for the core content, let alone optional secondary content like this.

The simple answer for this is probably just that the vast majority of people are not looking at the screen while listening, which probably makes it hard to make a business case for it to happen.

There is a id3v2.3 tag field "synchronized lyrics" (Field: SYLT)[1] where you can store a "phrase" with a "timestamp". One library I know that may support writing this tag is atldotnet[2], while I did not try it out. At least there is a Type TRANSCRIPTION in the linked code segment.

There is also LRC[3], a synchronized lyrics format stored in an extra file (*.lrc).

[1]: https://id3.org/id3v2.3.0#sec4.10

[2]: https://github.com/Zeugma440/atldotnet/blob/749d9ccb03032667...

[3]: https://en.wikipedia.org/wiki/LRC_(file_format)

Amazon's WhisperSync is pretty cool. I love listening to audio books but I also like to read and annotate them. WhisperSync solves for that. I don't buy any eBooks without that feature.

I do love that feature also. Sometimes I want to read read. Sometimes I want to read while driving. It's great.

https://www.snipd.com/ is another (automated - so it's not perfect) option

Another solution is Voice Dream Reader on iOS. I used to think I preferred an Audible audiobook because of the human voice. But when I want to take notes, I find I end up going back to Voice Dream. I even scan physical books to be able to read them in the app. Only for novels do I still prefer a human narrated book, provided there is a version with a really good narrator.

My experience with automatic transcription is that it introduces far more errors than it would eliminate.

If it’s automatic, I don’t see how it would be beneficial to listeners/readers. If it’s manual, it’s not cheap.

Audiobooks wouldn't need automatic transcription, they have the source book.

If I don't hear something - due to traffic or children or dogs barking, for example - automatic transcription still lets me pick up on some things even if it is wrong 25% of the time. In general, it usually isn't wrong enough that I can't get the context but instead, a word or two.

And if I'm already wrong, I know because of context and would hopefully pick it up.

In other words, it is still a benefit.

I don't know, I use it on YouTube a lot. It's pretty great.

Some sites do for podcasts.

But machine transcriptions are IMO useless for the purpose. (They're fine for interviews where they're just source material for something you're writing.) And they're especially useless for audio you're having trouble understanding anyway.

Human transcriptions aren't that expensive, about $1/minute, but they do cost money and you still need to spend some time cleaning them up and formatting them--and you still end up with words here and there you can't make out.

For audiobooks, there’s a legal question of whether it causes licensing issues. Audible got in a huge lawsuit over it which they settled out of court: https://amp.theguardian.com/books/2020/jan/15/audible-settle...

Audiobooks in the kindle app highlight the spoken text as the audio plays.

I believe that’s called a book, and you could use it to read along?

https://play.google.com/store/apps/details?id=com.google.aud...

My guess would be "it's expensive", same reason why live-action TV shows online didn't have subs until fairly recently.

A quick hack for fellow people who are hard of hearing: I often turn on a private "google meets" session, turn video off, and have google do the automatic captions through there if it's a captionless video I really want to see. I don't expect any privacy, and you do have to have a silent home for it, but hey. Works.

Search for "<podcast name> transcript". A lot of them do publish the full text of the show on their website.

Closed captions would be more convenient.

Closed captions make sense for video content because there are visuals on the screen to follow along with and associate with the text. For podcasts it'll simply be like reading a long article with one sentence displayed like every 15 seconds. If your eyes are anyways on the screen you'd rather turn off the audio and read it at your own pace.

Maybe you are doing something else like programming while listening to a podcast/audiobook?

It's still the same problem. Are you going to look at the code you are typing or the podcast text?

Transcripts largely exist for a different purpose than you seem to want.

They're typically a human-edited version of the podcast that's searchable, skimmable, and can be easily read instead of listening to the podcast.

Generally, when I'm listening to podcasts or audiobooks, I don't have a screen in front of me to put CC on. Might just be me, but I can only follow one source of information at a time, so I don't like non-music audio while using the computer or reading.

For podcasts only human transcription works good. Tech still is very hard on people with accents, even worse with non-english languages.

And human transcription costs like a lot 45 cents a word, imagine a 1 hour conversation.

Would something like this help? https://www.youtube.com/watch?v=pPBpxoIqDSA

On Android, you can turn on "Live Caption" for autogenerated captions a la YouTube

Nice project idea

rev.com

Is there an audio file format with teletext?

Reply on news.ycombinator.com