Whisper.cpp v1.4.0 | Hacker News Reader

Whisper.cpp v1.4.0

github.com

162 pointstosh45 comments

What’s the best-in-class Whisper implementation for real-time / streaming transcription? I’ve followed the various posts linked on this GitHub issue [1]; not sure if there’s more out there.

[1] https://github.com/openai/whisper/discussions/2

Works reasonable well in meetings.

https://github.com/davabase/whisper_real_time

Not Whisper per se, but IIRC LiveCaptions is partially trained using Whisper. It also works really well!

https://github.com/abb128/LiveCaptions

There is a Tensor Lite implemenation but it might not be maintained?

https://github.com/usefulsensors/openai-whisper

part of whisper.cpp:

make stream

./stream

This is really awesome. I hope that that at some point the WASM Whisper can be hooked up to WASM ffmpeg and let me drop audio files of any kind into that site.

https://whisper.ggerganov.com/

I released https://bigwav.app/ a few months ago, it does this.

This works really well! Thanks for making it and sharing. Hopefully you can find some income by offering the paid services!

FYI it seems to work on mobile sometimes but sometimes not at all. You might want to add a warning about when it might work or not work on mobile.

Any plans to make the source available for the client side part?

Thanks

Thanks, yeh at some point in the future I will likely release the source code, but it was quite a quick prototype so needs to be polished a little.

Mobile is an issue, some of the web APIs only work on desktop for now.

Thanks for the feedback

Are there any end-user friendly apps yet that make whisper accessible?

I’ve been playing around with various iOS apps today.

Someone else mentioned “Hello Transcribe”, which is a cool demo due to how real time the transcription is, but it’s effectively unusable for anything practical because it seems to split the audio on 30 second chunks (and not in the contextual way that Whisper normally does it).

Whisperboard is a free, open source iOS app (available on the App Store) that seems fairly user friendly, but there are a few crashes and bugs that the author is hopefully going to work out soon.

Aiko is extremely simple to use, but it only supports the “medium” model, so it isn’t very fast, but the results are good quality. Aiko is really designed to be paired with another app, like the Apple Voice Memos app, where you can share a voice memo directly to Aiko.

https://whisper.ggerganov.com/

does this count?

Oh nice, iirc it used to only have the smaller models there. Thanks for the pointer.

This one is excellent on MacOS: https://goodsnooze.gumroad.com/l/macwhisper

We make both Whisper and GPT-4 available in https://reflect.app

Another one for iOS though this one uses the small/medium models which definitely pushes the system to its limits: https://apps.apple.com/us/app/aiko/id1672085276

This mobile app uses Whisper: https://apps.apple.com/app/id6443919768

I’m working on a web app that uses whisper cpp wasm: https://www.transcriptey.com

It supports wav, mp3, m4a, and mp4. All the demuxers were written in JavaScript which was an interesting exercise. I couldn’t figure out a way using native JavaScript audio APIs to load small sections of long audio/video files without loading the entire file.

https://superwhisper.com

a macos menu bar app that uses whisper.cpp under the hood.

https://github.com/chidiwilliams/buzz

record a voice memo and receive it as email: https://whispermemos.com

Uses OpenAI API so really fast

These new models give me hope that computers will be able to understand me soon.

As a Flemish/Dutch speaker, not a single assistant is able to understand me in my native tongue. I can use Siri in English, but then she doesn't understand things like contacts or addresses. Also I cannot dictate text messages or people think I'm a spammer.

Tag v1.4.1 has been created, fixes bugs:

https://github.com/ggerganov/whisper.cpp/releases/tag/v1.4.1

Wow, I'm trying out the tiny 31MB WebAssembly model on https://whisper.ggerganov.com/ and it's working really, really well.

Is there a static library implementation with easy to use interface? Also think llama.cpp would benefit from having a lib release. It would be great to easily integrate both projects in existing c++ projects.

Exciting to see some parts now (optionally) running on the GPU.

Full circle eh. I wonder how well it compares to just trying to use the actual Whisper models on a variety of existing GPU capable bigger frameworks.

I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere. I expect Whisper.cpp probably better caters to users, is more readily consumable. Doing the same integer quantization that 1.4.0 whisper.cpp does would be another ask: how hard would that be?

Fwiw, Whisper.cpp uses Nvidia's cuBLAS. There does appear to be an AMD rocm port. https://github.com/ROCmSoftwarePlatform/rocBLAS

>> I don't know much practically about how hard it would be to take the Whisper PyTorch (1 or 2?) trained models & to make good use of them elsewhere.

What do you mean by "elsewhere"?

The whisper models can be used without much effort. The tooling provided by OpenAI allows for them to be used and installed in python with ease. People use them in companies to transcribe internal media datasets. No need for CPP implementation for internal uses.

Main problem is when _on device_ is required and the OpenAI model is not quantised and so use on say a smartphone becomes trickier.

Is there documentation somewhere on what the different quantisation methods are actually doing? E.g. what is Q4_0 vs Q4_1?

There's some info here: https://github.com/ggerganov/llama.cpp/issues/1240

how soon do you think these packages and a model will be included in modern OS?

Not soon enough, the iOS transcription is terrible compared to even the small whisper model. I really hope Apple revamps voice recognition this fall, but I'm not putting money on it being Whisper quality.

Probably really soon, especially on iOS/macos their implementation is awful.

Until then... https://superwhisper.com

I want to check it out but I'm reluctant when there aren't any screenshots / demos, etc. (Mentioning in case you're the developer)

Yeah I need to update the marketing site

Check out on https://twitter.com/superwhisperapp

I released https://bigwav.app, a web app that uses Whisper.cpp via WASM a few months ago.

Is what?

It’s an implementation of a speak to text software. Currently the best open source one, if not the best.

Particularly you can talk to it with a “international” English accent and it will work. Or French or Spanish or Norwegian… and it will transcribe English.

So could you use it to translate audio of a foreign language into english text?

Yes. According to the OpenAI Whisper repo [0], when you use the 'whisper' command line tool:

"Adding --task translate will translate the speech into English."

I tried translating a conference with a German speaker. The transcription was superb, but the translation no so much.

[0] https://github.com/openai/whisper

Ah I love the future. Thank you!

And OpenAI has a hosted version for just that https://platform.openai.com/docs/guides/speech-to-text/quick...

Yes.

Thanks for providing the missing info!

Yes, downvote a question that should've been answered by the title. Pathetic.

Reply on news.ycombinator.com