FOSDEM 2023 Transcribed by Whisper | Hacker News Reader

FOSDEM 2023 Transcribed by Whisper

jonatron.github.io

106 pointsjonatron48 comments

Open articleView on HN

It did not get confused by my heavy accent, and the result is surprisingly good:

https://jonatron.github.io/fosdem2023whisper/files/fast_data...

Original: https://www.youtube.com/watch?v=JlcI2Vfz_uk

But some mistakes are hilarious!

> 500 megabytes per second, Britain.

Britain. Guess what it actually was :)

To be fair, you did say Britain, when I think you mean B Written.

This is pretty good! I spot checked one video (https://jonatron.github.io/fosdem2023whisper/files/rust_core...) and it's done a decent job. It mostly makes mistakes around a few proper nouns.

- coreutils => corridors/coroutils/cori-teals/curricules/corretails

- Rust => rest/Resi

- glibc => GLC

- Gentoo => Gen2

- Fuchsia => Fushia

- SELinux => AC Linux

Other proper nouns it transcribed well - LLVM, Mozilla, Debian, Ubuntu, clang. These are much better known I suppose.

It finds the speaker's French accent a bit difficult, because I see a few mistakes like "find => fine", "imposter => imposterous", "command => comment".

Overall pretty good. The only thing I'd like to improve readability is a pass with an LLM to break it into paragraphs but that would be way more compute and no longer local (like Whisper).

Maybe fine tuning on "tech lingo" would improve that. Give that to your average person to transcribe and the results most probably would be worse than that.

I was wondering how that compared to Vosk, a lightweight pip-installable speech-to-text library. So I ran Vosk on some of the videos to compare. The model I used is 40MB and transcribed the 32min keynote in 2m12s on 1 CPU core, using 400MB of memory.

The result is not that far. There are some errors of course, notably on acronyms and technical terms ("IBM", "GNU", "GitHub"), but a lot of the mismatches are in areas where both models got things wrong.

Neither of the models got "FOSDEM" right, though Whisper is consistent at using the word "fosstem", whatever that is?

Very Nice. I've use the "all transcript" to make a Word Cloud: https://framapiaf.org/@ldubost/109851711189159771

Wow!

In terms of resource usage, for this and many other "emerging" opportunities, is there some thinking about pooling idle CPU's/GPUs for creating open and widely available models and/or datasets based on models?

SETI type crowdsourcing of compute was always a bit niche but maybe now we have some use cases of very broad interest

The New Yorker had a recent piece on Whisper as well.

https://www.newyorker.com/tech/annals-of-technology/whispers...

For them, the key difference (other than it's great performance) was that it was a standalone executable that did not need to access remote resources or "share" sensitive information with such remote resources.

Doesn't Mozilla have their own speech recognition library? I think it would have been more interesting to see the results, since we're talking about FOSDEM.

Also, since OpenAI was already used, it could have been cool to provide a summary of the transcript!

whisper is mit licensed.

and run "offline" and locally, so the content transcribed is not sent anywhere.

There is also a CPU only version discussed a few times: https://news.ycombinator.com/item?id=33877893

> Also, since OpenAI was already used, it could have been cool to provide a summary of the transcript!

I liked this idea, so I opened a PR for exactly that: https://github.com/jonatron/fosdem2023whisper/pull/2

Missed opportunity to feed the transcripts to LLM and prompt it to clean it up, (remove stop words, stuff like that)? -> instant blog post to go with the talk.

I tried to get chat.openai.com to generate something interesting for the transcript linked to in another comment and it failed rather spectacularly. The transcript contains error for Rust (which becomes rest) and coreutils (which becomes corridors) making the output of the LLM useless. The LLM also seems to completely misinterpret the author when he speaks about working for Mozilla, omitting the part where he explicitly mentions that he's only saying this so that you wont associate this work with Mozilla. It also repeatedly gets confused about what the author is talking about, likely due to a small context size.

It would be a very low quality blog-spam level blog post.

I'm curious if you could use the transcript + something like the title/subtitle of the talk with chatgpt to fix the spelling mistakes

Interestingly, If i give it the filename title "Rust Coreutils" it spots "the corridor -> coreutils", but if i provide the proper title of the talk "Reimplementing the Coreutils in a modern language (Rust) Doing old things with modern tools" it doesn't find any mistakes. Once again lending creadence to the observation that it doesn't actually "understand" since the proper title is quite clearly not markedly different (in relation to the transcription mistakes) from the filename title.

I had something more basic in mind: Just clean up syntax without much, if any, semantic re-interpretations.

Subtitles, LLM, DeepSpeech comparisons are good ideas, so say if you're going to do them. I'm not going to do much more.

You can git pull from here for subtitles in the WebVTT format: https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt

(I'm not able to make a pull request because GitHub is giving me the pink unicorn error page when I try to fork your repo.)

Got a reply from GitHub that they fixed the problem, so here's the pull request: https://github.com/jonatron/fosdem2023whisper/pull/1

This is great! Does it include all talks? I just noticed that it might be missing https://fosdem.org/2023/schedule/event/nix_and_nixos_runix/.

The accuracy of whisper seems to be the best available for mere mortals, but the resource requirements are fairly high. It doesn't seem like it is possible to use it for real-time transcription.

What solutions are Twitter spaces and the like using for real-time live captions?

there's a c++ fork of whisper that can run the smaller models on a laptop CPU in real time

Looks like it has improved a lot since I last checked it out.

Do you happen to know what kind of EC2 instance (or equivalent config) would be required to run it?

I checked my own talk and it's not perfect but it's more than understandable, despite my having an accent, speaking fast and not having rehearsed very much. Together with the slides it would be clear what was going on.

I wonder how hard it would be to automatically add slides by scanning the talk video for points at which the on-screen slides change and mixing a frame from the video into the transcript at the relevant point.

1. Automatically detect when (timestamp) change in slide, e.g., large % of colour change in the image of the video (I assume video is slide-deck only else we can crop to the area that is);

2. Use the timestamps provided by whisper to group and align with (1).

I don't know how companies like Grammarly will survive this AI summer.

This is wonderful. I was able to search for my favorite topics by cloning the repo and grep through all the TXT files. Amazing!

Every use of "fuck":

> So, you can see my Firefox is in Italian, but you can see that it automatically detected that the page is in French and it is suggesting me to translate it to Italian. I will change it to English. Oh, fuck.

> The reason for that is because there is so many ways to extend or to define an architecture, and there is some really fucked up things that can be done in so many architectures

> We don't measure RSS because it measures anything useful we measure it because it's really fucking easy to measure.

> There were a few customers that didn't respond, but at some point you just have to decide to. Don't give a fuck.

> Computational storage, what the fuck's that?

> like on MacOs you can have a recovery key, on Windows you can have a recovery key, and this is what we should do here as well, right, like use TPMs absolutely, and then hide entropy recovery keys that things are fucked up, things are fucked up

> we get money, we invest in open source, and then we have the problem that what we do in open source could be reused by non-European companies. And thinking, oh, my God. Oh, that's so fucking wrong. Why do you do that?

> So, you know, if Monty is not screaming at me saying you are fucking moron, Peter, that is not how it is, then probably I am not doing my job properly.

> I'm going to wail and gnash my teeth and hope it doesn't happen in practice because at the moment, you know, something like EDK2, to be clear, by the way, I don't give a fuck about non-free firmwares

> Is there anybody here who doesn't know what a kubelet is? Okay. The kubelet, oh, you fuck, anyway.

> So for now we're targeting your own CA so you can just say fuck it, I will enroll my own keys to the firmware.

> So one of the idea is to start to come to a more predictable fashion, which is one major API break and API break every year around December, January, so we're in February, and we fuck this year.

> It's really not sure whether we are going to be able to do that because you're transpiling assembly. Like, what the fuck are you talking about?

> your problem was where you called, in this case, pop on an empty stack, and not the completely pointless unwrap in there, because you want to know where you're fucked up and not here, obviously.

> So if it is, then say, say like, this is fucking wrong, Peter, you know, so I can fix my slides when I talk next time, I have the wrong stuff, right?

OOps. Busted :) At least one of those is mine

Can you describe your setup / how you piped the audio from YouTube to Whisper in order to generate these transcripts?

Used a small python script to download the webm files from https://fosdem.org/2023/schedule/events/

Used a small bash script to run `whisper $f --output_format json --language en | tee $f.txt` over every file.

Ran a small python script to make the HTML. All totally trivial, it just required some GPU-hours.

Thanks, that's very helpful. (I hadn't realized that their native utility used ffmpeg and could decode basically any audio because of that -- very cool!)

How heavy on resources is Whisper? How long does an average GPU need to transcribe, say 2 hours of video?

Wow! Is there an easy way to combine these subtitles with the video in the browser?

Not a format I would recognize, but I am not an expert in the area. Two lines of awk would convert it to srt I guess.

It's even closer to WebVTT, just had to add the line "WEBVTT" to the top and replace all square brackets with line breaks:

    curl https://jonatron.github.io/fosdem2023whisper/files/celebrating_25_years_of_open_source.webm.txt | ( echo "WEBVTT"; sed -e 's/^\[\|\]  \?/\n/g' ) >celebrating_25_years_of_open_source.webm.vtt

Plays in the browser with HTML like this:

    <video
        controls
        src="https://video.fosdem.org/2023/Janson/celebrating_25_years_of_open_source.webm"
    >
        <track
            default
            srclang="en"
            src="celebrating_25_years_of_open_source.webm.vtt"
        >
    </video>

A sidenote, but adding subtitles to pure audio in HTML is surprisingly difficult!

Does it not work to put a track element inside an audio element?

You can try to fake it as a video, but in my experience, it is difficult to properly format the element so that the subtitles or captions actually shows

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/au...

> The <audio> element doesn't directly support WebVTT. You will have to find a library or framework that provides the capability for you, or write the code to display captions yourself. One option is to play your audio using a <video> element, which does support WebVTT.

Is my version of sed dumb, or is 's/^\[\|\] \?/\n/g' effectively a no-op? Or did HN garble something?

I used GNU sed on Ubuntu. The only difference I notice is that you have one space where I had two, but that difference shouldn't matter actually. Perhaps your sed has a different opinion on which of the backslashes should be there.

You can access the resulting WebVTT files here: https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt...

Once again macOS non-standard utilities bite me.

     -E      Interpret regular expressions as extended (modern) regular expressions rather than basic regular expressions (BRE's)

    -r      Same as -E for compatibility with GNU sed.

Thanks for providing the complete set of vtt files. I found some interesting transcription errors in the talk on Elliptic curves in FOSS. For example "jealous two curves" instead of "genus two curves" and "ethyl curbs" instead of "elliptical curves"

It looks pretty easy, but I'm not going to do it, if someone else wants to.

This is gold for us not native english speakers.

It's gold for non-native speakers of any supported language.

I'm trying to learn some Spanish, and a fun and easy way to study is to watch films in Spanish with Spanish-language subs.

Finding Jurassic Park on piratebay was easy, finding the right subs was impossible. Enter Whisper.

My favourite little hack this year.

Uh, nice, thanks for sharing!

Reply on news.ycombinator.com