VLC tops 6B downloads, previews AI-generated subtitles
techcrunch.com
techcrunch.com
I’d love a feature like that for Jellyfin eventually.
Automation I suppose. I only run Sonarr/Radarr, so two additional services. They both work exactly the same, so it's basically "configure once, run twice".
I eventually ran out of patience to find the right releases and what not, and now enjoy a life where I can just input the title, then return one hour later to everything setup for me. Any new episodes/movies will just be available automatically in Jellyfin next time I open it up on the TV. Helps that other people in the household are also able to use it, so we've got rid of all streaming services now.
1. Overseerr - add tv/movies in one place 2. Custom feeds as import lists so that new, popular stuff is automatically getting added 3. Kometa (for Plex users) - custom collections via trakt and burning imdb/rt/metacritic scores into cover art
What do you mean with this? Are you automatically downloading things based on popularity solely, or does this mean something else?
Please no. Some subtitle companies do think like this, and it's really weird, like when they try to "convert" cultural jokes, and then add in a bunch of more assumptions regarding what cultures you're aware of depending on the subtitle language, making it even harder to understand...
Just because I want my subtitles in English, doesn't mean I want all typical discussed Spanish food names to be replaced by "kind of the same" British names, yet something like that is something I've come across before. Horrible.
Jokes are an example of something translators can do really well - things like puns don't work 1:1 across languages. A good translator will find a corresponding, appropriate line of dialogue and basically keep the intent without literally translating the words.
Food is kind of silly because it's tied to place - if a setting is clearly Spanish, or a character is Spanish, why wouldn't they talk about Spanish food? Their nationality ostensibly informs something about their character (like Godzilla) and can't just be fine/replaced.
Again, those aren't "cultural translations" but "idioms translations", which I do agree should be translated into something understandable in the language, otherwise you wouldn't understand it.
What I was aiming at in my original comment was examples like these:
> Family Guy original voice-overs + subtitles making a joke about some typical father figure in Hollywood for example. Then the Latin Spanish subtitles will have translated that joke but replaced the specific actor with some typical father figure from the Mexican movie industry, while the Castilian subtitles would have replaced it with someone from the Spanish movie industry.
His translations were nowhere near what the movie was about, but they were hilarious and fit the plot perfectly.
On the other hand, his Lord of The Rings was an "alternative" dub as you described. Didn't watch that one though.
The purpose of a translation is after all to convey the meaning of what was said. So for example you'd want the English "so so" to be translated in Spanish as "más o menos" instead of repeating the translation of "so" twice. You don't want to just translate word for word, venir infierno o alta agua.
A lot of dialog needs language specific context, many expressions don't lend themselves to literal translation, or the translation in that language is long and cumbersome so paraphrasing is an improvement.
Like with anything else, the secret is using it sparingly, only when it adds value.
Agree, but I don't think those are "cultural translations" but more like "idioms translations", which mostly makes sense to do.
What I originally wrote about are things like Family Guy original voice-overs + subtitles making a joke about some typical father figure in Hollywood for example. Then the Latin Spanish subtitles will have translated that joke but replaced the specific actor with some typical father figure from the Mexican movie industry, while the Castilian subtitles would have replaced it with someone from the Spanish movie industry.
Without adapting the translation the natives will immediately understand the reference while you, the non-native with no sense of who they're talking about are left wondering what's the real message. Calling someone a "Mother Teresa" might miss the mark somewhere in China. Same if an Italian movie made references to food like Casu Marzo and the average American would probably miss a lot of the context.
Just recently I saw this in a series where some workers in the oil extraction industry were staring at a pan of paella asking what it is and calling it jambalaya. Paella is world famous, how about khash?
That's why I said I understand the usage but it should only be done if it really helps the comprehension, not just the principle of gratuitously adapting to one language.
Right, but isn't that why the American is watching this Italian movie anyways, to get a wider understanding of Italian culture? I don't watch foreign movies with the expectation that they're adapted to my local culture, then there wouldn't be much point in watching it.
I don't know. I always though people would prefer to hear the original voice of the actor, since speech is a big part of the acting. But movies are dubbed in a lot of countries. And most people from those countries I spoke to said that they find the idea of subtitles very odd because it's distracting them from the movie.
I am sure many if not most people simply want to understand what's the message behind the conversation on the screen first and foremost. Learning something new only works if while you try to keep up with the dialogue, you also keep track of all the expressions you heard for the first time, to look them up later.
This is why I think it's the translator's job to balance translation and adaptation. Directly translate the original where context helps you understand the meaning so you get the "original" experience, and adapt where leaving just the 1:1 translation will make you lose the thread or miss some details.
They're using Whisper for speech to text, and some other small model for basic translation where necessary. It will not do speaker identification (diarization), and certainly isn't going to probe into narrative plot points to figure out if naming a character is a reveal. It isn't going to place text on the screen according to the speaker's frame place, nor for least intrusion. It's just going to have a fixed area where a best effort at speech to text is performed, as a last resort where the alternative is nothing.
Obviously it would be preferred to have carefully crafted subtitles from the content creator, translating if the desired language isn't available but still using all the cues and positions. Secondly to have some carefully crafted community subtitles from opensubtitles or the like, maybe where someone used "AI" and then hand positioned/corrected/updated. Failing all that, you fall to this.
That's just bad destructive art, especially for a foreign language that you partially know.
> Sometimes you need to name a voice as unknown, to avoid spoilers.
Don't name any, that's what your own eye-ear voice recognition/matching and positioning are for (also reduces the amount of text)
> Sometimes the positioning on the screen matters.
This is rather valuable art indeed! Though unlikely fit to be modelled well
That’s tricky when one or more speakers aren’t visible.
You can probably get 99% there without that for a lot of content, but I'd challenge the notion that this is somehow only important for hearing impaired viewers (or people just watching without clearly audible sound for other reasons).
I don't think that's an unrealistic goal to have for AIs; they're already extremely good at semantic scene description after all. By looking at the image in addition to just the audio track they probably also get a lot more metadata, which a refined world model will eventually be able to use just like a human subtitle editor can today.
There is no need for artistic interpretation, substituting words, or hiding information. If it’s in the audio, there’s no reason to keep it out of the subtitle.
An AI subtitle generator that takes artistic license with the conversion is not what anyone wants.
Questionable. It drives me crazy to have subtitles that are paraphrase in a way that changes the meaning of statements.
I really hope people to stop doing that.
So, knowing exactly the words being said is of utter importance.
NO!
I speak and understand 90% of English but I still use subtitles because sometimes I don't understand a word, or the sound sucks, or the actor thought speaking in a very low voice was a good idea. When the subtitles don't match what's being said, it's a terrible experience.
Pretty sure this is a violation of the Americans with Disabilities Act, so illegal in the U.S. at least. Being Deaf doesn't mean you need "reduced" dialogue.
Agreed.
> Sometimes it's better to paraphrase dialog to reduce the amount of text that needs to be read.
Hard no. If it’s the same language, the text you read should match the text you listen to. Having those not match makes parsing confusing and slow.
> Sometimes you need to name a voice as unknown, to avoid spoilers.
Subtitles don’t usually mention who’s talking, because you can see that. Taking the source of a voice is uncommon and not something I expect these system to get right anyway.
The experience wasn't really good to be honest - the text was technically correct but the way multiline phrases were split made them somehow extremely hard to parse. The only reason I learned AI was involved is that it was bad enough for me to stop viewing and check what was wrong.
Hopefully it's an implementation detail and we'll see better options in the future, as finding subtitles for foreign shows is always a pain.
VLC has the option to find subtitles. If you use plex or jellyfin there are add-ons or bazarr, which does it automatically
- Sometimes the creator bases their captions on the script and misses changes in edit
- Sometimes the creator's captions are perfect transcriptions but broken up and timed awkwardly
Auto-generated captions aren't always perfect but unlike human captions provide word-by-word timing.
(By "connecting tissue", I mean they don't have the problem where sentence A is "I like chocolate", sentence B is "only on special occasions", and at the time B appears A is completely gone, but you really need A and B to be onscreen at the same time to parse the full meaning intuitively.)
> Man Builds Background of Tan for Swedish House
It's completely puzzling. To understand it you have to know both English and Romanian. "Background of tan" is the term for "foundation" (makeup) in Romanian. That is, "foundation" has two meanings in English: for a house and for makeup, but Google uses the wrong one.
Automatic translation is full of these bad associations. I have no idea how people who don't speak English understand these translations.
Maybe they're really using a truly open source model (probably not) but the meaning of the word is muddied already.
Here they are working on integrating Whisper.cpp
In the search bar it says "Updated 2 weeks ago", like if there were additional recent comments or actions in this thread that we cannot see.
So it could actually be OpenAI Whisper model, for which we have the final binary format (the weights), but not the source training data, but it is the best you can get for free.
As another way to look at this, where does this questioning of energy use end? Should I turn off my laptop when I go to the supermarket? When I go to the toilet? Should I turn off my lights when I go to the toilet?
My point is, we do a lot of inefficient things and there is certainly something to being more efficient. But asking “is it efficient” immediately when something new is presented is completely backwards if you ask me. It focusses our attention on new things even though many old things are WAY more inefficient.
That's what a software engineer would say who views resources as unlimited and free.
from the same guidelines: don't be snarky; don't post thoughtless insubstantive comments; reply to the argument instead of calling names; don't sneer; respond to the strongest interpretation - assume good faith; eschew flamebait; don't trample curiosity.
You're making a litany of unsolicited accusations while subtracting value in a conversation you're not a part of. Please stop with the self-appointed fixation on microaggression policing because it just creates pointless drama. Thanks.
Today we consume twice as much energy as we did in the 80s (and that's mostly coming from an increase in fossil fuels consumption). Datacenters alone consume more than 1% of global energy production, that doesn't include the network, the terminals, and the energy necessary to produce all of the hardware.
Energy-use or electricity-use? Only about 20% of total energy use is electricity [1]. I went through the math for EUV machines in more detail at https://news.ycombinator.com/threads?id=huijzer#42600790.
[1]: https://ourworldindata.org/energy/country/united-states
Who's "we"?
Worldwide we've gone from 4.4 tons/person to 4.7 tons/person (7% increase) from 1980 to 2023: https://ourworldindata.org/grapher/co-emissions-per-capita?t...
Because today is today, and if we can project that the energy consumption of doing a task n times on the client side outweighs the complexity of doing it once and then distributing the result somehow to all n clients, we should arguably still do it.
Sometimes it's better to wait; sometimes it's better to ship the improved version now.
> Computation is really negligible in the grand scheme of things.
Tell that to my phone burning my hand and running through a quarter of its battery for some local ML task every once in a while.
Yes.
The results could be cached, but it's probably unlikely that they would need to be used again later, as I imagine most videos are watched only once.
Another option would be to upload the generated subtitles to some service or p2p, but I believe that would also problem (e.g. privacy, who runs the service, firewalls for p2p, etc).
https://github.com/videolan/vlc/blob/f908ef4981c93a8b76805ad...
and to their own servers:
https://github.com/videolan/vlc/blob/f908ef4981c93a8b76805ad...
should could fetch subtitles as the same time ?
edit: cf, what "a3w" says too.
You can navigate to $foo.exampl.page and it will generate a website on the fly with text and graphics using AI. It will then save and cache the page.
It’s admittedly a useless but cool little demo.
So while I'm excited this feature is now available, having high quality subtitles cached in one place and generated by AI is the answer imo.
Previously, that was used for mp3 album covers or something?
I don't know how many times I've seen subtitles that appear to be based on a script or were half-assed, and don't match the dialogue as spoken at all.
But yes, that would be quite nice, if there are enough people who don't mind uploading the file names of the media they play (or some other unique media identifier?) along with the subtitles to that service—and with user credentials? It would certainly need to be opt-in, which makes one wonder if it would be much effective at all.
Go-to player with easy wifi loading and ability to connect to file shares to find files. Simple and actually easy to use (of course having a file server is another question)
In the ideal case you'd probably generate a first pass of subtitles with AI, have a human review and tweak it as needed, and then share that. There's no reason for people to be repeatedly generating their own subtitles in most cases.
You also want local generation in a lot of cases. If you have your own videos you need to generate them yourself. For Accessibility it's fantastic if they can have subtitles on every Video.
If generating your own is fast and good enough and takes little compute, then it isn't needed to share them. Having subtitles generated by the best models and optimized by humans is better, but not needed in most cases.
VLC: we're gonna work on AI
Dude you need to level up your reasoning skills.
VLC is a good example of an OSS project that is pretty well run with decades of history that has a healthy ecosystem of people and companies earning their living supporting all that and a foundation to orchestrate the development. I don't think there ever was a lot of VC money they need to worry about. This is all organic growth and OSS working as it should.
So, this boils down to what paying customers of these companies are paying for. The project also accepts donations but those go to the foundation and not the companies. It's the companies that employ most of the developers. And you can't fault them on working on things that they value. If AI features is what they pay for then that is what they work on.
I happen to share your reservations about the UX. It's a bit old school, to put it mildy. And they obviously don't have professional designers that they work with. Like many OSS products, it looks and feels like product made by techies for techies. It doesn't bother me that much but I do notice these things. I actually talked to one of their IOS developers a few years ago. Pretty interesting person and not a huge team as I recall. I remember talking to her about some of the frustrations she had with the UX and the lack of appreciation of that. I think she moved to Netflix afterwards.
Like with most OSS projects you are welcome to take part in the meritocracy and push your favorite features or pay somebody to do that for you. But otherwise, you should just be grateful for this awesome thing existing and prospering.
That's not the problem. mpv is another media player that is arguably even more "made by techies for techies", yet it doesn't have the usability issues of VLC, and is a much more robust piece of software.
VLC is just poorly designed from the ground up, and the project's priorities are all over the place, as this AI initiative demonstrates.
It's common with many OSS projects. There are a few positive exceptions. But this stuff is hard.
I've always found their quality to be somewhat lacking.
Likely due to cost, Google hasn’t decided to use any LLM model (what basically everybody refers to when they say AI now) to generate the subtitles and by modern standards they aren’t very good.
(Like open source software, so that, in theory, someone could see see the source code, source data, and process for how it was trained, and reproduce that? And could they change how that model is built, not just tuning atop it, and distribute that under the same open source license?)
It's funny for a second, and then incredibly frustrating after that. Character names are regularly wrong, and sentences get rewritten to have different meanings. Instead of just reading subtitles, you have to mentally evaluate them for errors.
At best, AI generated subtitles should be a last resort.
Who in their right mind would aspire to become a professional translator today?
> Many anime fans were concerned about the low-quality subtitles in The Yuzuki Family's Four Sons' debut episode, leading some to believe that they had been generated by A.I. These subtitles were notoriously awkward -- occasionally to the point of being nonsensical -- and featured numerous grammar issues; Crunchyroll was forced to delete the episode following outrage by fans on social media, with many asking why they didn't pay professionals to do a better job.
https://www.cbr.com/crunchyroll-ai-anime-subtitles-investmen...
Further thoughts on the matter: https://old.reddit.com/r/grandorder/comments/dnpzrh/everyone...
DeepL is already pretty good, but it still needs a proper translator for the optimal output. That translator just saves a lot of time not having to translate every word.
Obv you can find majority of subs online for popular files but niche cases usually have low subs available and sharing AI could help the ecosystem.
Initial versions will be more power hungry, but it'll be negligible at the end, given that modern processors have accelerators for image and voice based AI applications.
https://support.microsoft.com/en-us/windows/use-live-caption...
To be fair, technically speaking this isn't nearly as easy as step forward due to how codecs are constructed, but lots of players have this feature already.
That's not frame interpolation. Just being able to step back one frame at a time.
this isn't the bad kind of ai we hate... it could be actually useful!