Faster Whisper Transcription with CTranslate2 | Hacker News Reader

Faster Whisper Transcription with CTranslate2

github.com

46 pointsLwrless15 comments

The project page mentions whisper-diarization (speaker recognition) as a user of faster-whisper. I've been in the market for that, definitely going to try it out.

https://github.com/MahmoudAshraf97/whisper-diarization

There's also https://github.com/akashmjn/tinydiarize which received support recently in core whisper.cpp - it doesn't support the large model yet, but the results are promising

That’s a pretty exciting development. Is there any putative timelines on when it might get expanded to the large models? At this point, I would have to do a two pass workflow: first with large for accuracy and a second with the tube to assign speakers.

Not that I can see, the developer's roadmap[1] currently is at writing a blogpost about it & trying different sampling methods, expanding to the large model looks to be a long way off (which is a real pity, even if there was just a finetune of large for english that would be a big help over the existing small english finetune).

Could you go into more detail about your workflow? I'd been considering a two-pass approach myself until I discovered tinydiarize mentioned in whisper.cpp's --help text

1: https://github.com/akashmjn/tinydiarize#roadmap

I have not yet done this, but my thinking is that the diarization process would be useful data, but not so much that I would be willing to accept the quality dip of using the tiny trained model. I already run the large model at the word level, so I have high quality transcription + timestamps. Then I would perform the second whisper-pass with the diarization and discard all of the output except when a speaker change is identified. Merge the two results based on timestamps and there you go.

Interesting - I'd been considering basically that two-pass approach in reverse (diarize, isolate each utterance in a per-speaker file with a vocal separator between each clip so that I don't have to rely on timestamps being perfectly accurate, run through whisper - could also do a file per utterance, but setup time seems like it would dominate in that case). Concern I had about my approach was that it'd be too different from whisper's training set of conversational data, and that might cause accuracy to take a dive.

Please let us know how it goes. I struggled to get speaker diarization working on an M1. Need to revisit.

Has someone already compared to whisper.cpp running on the CPU with cuBLAS on the GPU? That was a lot more than 4x faster than whisper on the CPU, and was sufficiently parsimonious with GPU ram to allow me running even the large model (and without quitting firefox, which is a bit of a GPU ram hog).

Hopefully someone also makes it useful and ports it to a c/c++ lib like whisper.cpp

Why is it useless if not?

It is only useful if you are shipping a python app. But you are not doing that on mobile or the vast majority of desktop apps.

Ate there any plans / usage examples of using this behind a Web app (node/react is my go to stack at the minute).

I'm currently using openai whisper rest API, but sporadic latency spikes are a problem.

Can someone try to explain this architecture / whisper reimplementation? Thanks.

The original Whisper implementation from OpenAI uses the PyTorch deep learning framework. On the other hand, faster-whisper is implemented using CTranslate2 [1] which is a custom inference engine for Transformer models. So basically it is running the same model but using another backend, which is specifically optimized for inference workloads.

[1] https://github.com/OpenNMT/CTranslate2

Much appreciated.

Reply on news.ycombinator.com