Hacker News Reader

Comment by peterburkimsher | Hacker News Reader

The article says that "Subtitle files... are not encrypted". Netflix subtitles are really interesting to me, because they're aligned in different languages. A lot of unofficial SRT files have different timestamps.

The reason is that I'm trying to collect data for learning languages, especially Chinese. Kind of like VoiceTube [1] and Youglish [2] are doing for English. So far it's only for personal use, not a revenue-taking company.

I tried investigating a Netflix stream, but the subtitles weren't plain text! They were distributed as PNG files [3].

Does anyone have more information about this? Do the pirates really OCR the subtitles for their MKV files? (I doubt it). Is there another way to get the plain text? Contact me directly if you're afraid to comment publicly.

[1] https://www.voicetube.com/

[2] https://youglish.com/

[3] https://www.slideshare.net/RohitPuri23/timed-text-at-netflix...

View on HN

> The reason is that I'm trying to collect data for learning languages, especially Chinese.

Subtitles sites (opensubtitles, podnapisi, etc.) have similar data and simple alignment algorithms will work for unaligned subtitles.

> Do the pirates really OCR the subtitles for their MKV files?

OCR is really simple for text. No reason why they wouldn't do it. There were free programs decades ago for OCR when you were ripping DVDs and they worked mostly on character patterns (you would start with an empty database and then just tag the pattern, after a minute you'd have the whole subtitle OCR-ed).

Please can you recommend a "simple alignment algorithm"? I can send some example ENG&CHT subtitles your way. There's a different number of subtitles, shown at different times, breaking lines in different places. It's a mess. If you know how to align it quickly, please tell me! Doing it manually takes me a couple of hours: longer than the whole movie. That's why I've only done it for a very few, specifically for movie nights with friends.

You need good dictionaries, giving you all possible translations of a word both ways. Given that subtitles in both languages will have matching sentences very closely you can easily determine (be it some counting criteria, simple linear model or dynamic programming) which sequence of let's say 3 subtitle lines matches a sequence of 5 subtitle lines in the other translation.

People/place/town names align stuff pretty easily. The fact that a sequence of lines matches some other sequence of lines makes finding the optimal alignment very efficient too.

I'm guessing that just making a set of words for each subtitle line, counting the common words and picking a criteria of deciding if line maps to other line is more than enough. Subtitles are much easier than free-flowing documents because they are time constrained.

Please show me your best transcriptions for this channel [1]. There's over 2 TB of video data. No way to get access to the original PPT files.

At least the subtitles on there are already aligned English & Chinese, but they do require OCR. Message me directly if you want my OpenCV script that can pull the yellow text out.

Tesseract gave me this. The W0「d n! Gnd has bP〔‥mE ‥任Sh 嬰孩降生道咸肉員 Peace has Come for O… Km… 峒﹏m US 靜安來自‵我君手 `

I'm sorry to continue insisting, but seriously, Tesseract gave me terrible results. Identical lines don't even give the same OCR result, so trying to run partial matches on Google Translate-quality word equivalents isn't going to cut it. The OCR can't even give me the right number of characters! In the end I'm trying to do it manually, but it's really time-consuming.

[1] https://www.youtube.com/channel/UC8_emPVKZvOwJnoBVpU0ytg/vid...

You should not use Tesseract for subtitles that blend with the colorful background. Some leptonica magic can extract that yellow stuff pretty easily. OpenCV is not as powerful for it, although it does have some leptonica functionality.

I'll try to make a simple one in the next couple of hours.

Although, the task you're asking me to do is different from subtitle alignment you could do on .srt files.

What you can do with leptonica is extract the symbols (letters and ligatures) and manually tag them. You could also extract the lines containing letters and then process them through tesseract.

Poured out on the feet of Jesus

我們傾倒耶穌腳前

This is my output from tesseract (chi_tra) (for https://www.youtube.com/watch?v=RzIh2pamcwU)

I have just extracted the U component (YUV colorspace), binarized the image and tesseract just loves it. The players in the background are completely removed.

I would probably use some background removal or some morphological operations from leptonica to make stuff more robust (instead of binarization).

Of course, if tesseract does something incorrectly you can do a quick connected components in leptonica, extract the symbols and do your own OCR (manually). For chinese it's a bit of an issue because there are bunch of symbols, but for english you will be done tagging all the unique symbols in no time.

That's amazing. Please can you tell me which version you're using? I tried

tesseract 3.04.01 leptonica-1.74.1 libgif 4.2.3 : libjpeg 9c : libpng 1.6.34 : libtiff 4.0.9 : zlib 1.2.11 : libwebp 1.0.0 : libopenjp2 2.3.0

If you could email me your scripts, that would be extremely useful. And then I'll send you some subtitles where alignment is important. I just used this channel as an example of where I'd tried Tesseract and given up.

This tool https://github.com/kaegi/aligner worked well for EN-JP subs when I used it. The guy who wrote it describes the algorithm. Hope this helps.

This is similar to idx/sub files from DVD which are just images and they are OCRd to text sub/srt file. I assume it's just easier for them to render the image than to mess up with text rendering/code page issues/whatever complication coming from huge diversity of devices they support.

>Do the pirates really OCR the subtitles for their MKV files?

Yes. At least as far as I have seen, which is not scene level but in private tracker discussion forums.

OCR-ing digital text is pretty accurate afaik.

OCR-ing ASCII text is bearable. I've tried Tesseract and get about 80% accuracy for English. But not Chinese. That's about 40% at best. The only decent OCR for Chinese I've used is Google Cloud Vision, and that costs money.

https://nikse.dk/SubtitleEdit uses tesseract as one method for OCR. Sometimes there are errors, like I and l but there is a "correct common errors" mode you can use.

Please can you help me try to extract the subtitles from this channel? [1] Tesseract gave me this. The W0「d n! Gnd has bP〔‥mE ‥任Sh 嬰孩降生道咸肉員 Peace has Come for O… Km… 峒﹏m US 靜安來自‵我君手 `

That kind of error ratio isn't in the realm of "correcting common errors". Surely there must be better software, but I can't find anything that runs locally and can read these subtitles. I've resorted to typing them out by hand, which is super slow because I don't yet speak Chinese, so I have to use handwriting recognition to enter each character.

[1] https://www.youtube.com/channel/UC8_emPVKZvOwJnoBVpU0ytg/vid...

You probably need to preprocess the subtitles before feeding it to Tesseract. Crop them to size, use some thresholds to only get the subtitle in the image. Seperate by languages.

> I'm trying to collect data for learning languages, especially Chinese

Viki has a lot of subtitled Chinese shows. They generally don't offer Chinese subtitles for the Chinese shows though.

The show itself often will, but those subtitles are just part of the video, not even an overlay.

Yeah, I really need to be able to grep for a specific word to make videos like this:

https://www.youtube.com/playlist?list=PLhIooD7mFhphhT5nDdhK0...

I wonder if Netflix’s use of image subtitles is a carryover of how subtitles work on DVDs — images as well.

It's not a carry-over from their DVD business. It's a consequence of how they choose to stream. My assumption is that image-based subtitles are more compatible across a range of devices and clients, as you get to choose how they look beforehand.

You can definitely send subtitles as raw text and metadata – it's what they do on iOS devices IIRC, since you can change the subtitle appearance on the client.

Also, their ingest works with TTML (a text-based format): https://partnerhelp.netflixstudios.com/hc/en-us/articles/215... – this is a fairly widely used standard.

Reply on news.ycombinator.com