Distil-Whisper: distilled version of Whisper that is 6 times faster, 49% smaller
github.com
github.com
Should make running in the browser feasible even for underpowered devices: https://whisper-turbo.com/
[0] - https://github.com/OpenNMT/CTranslate2
[1] - https://heywillow.io/components/willow-inference-server/
It does speech recognition
I guess it could be worse. In the future our ai overlords will just talk in embeddings (not even abbreviations), and we will have no clue what they are talking to each other about.
Informally, it feels odd/wrong to see people acting like it's a gap. I've been swimming in the AI stuff for a year so I chose to frame it as cultural mores.
I don't know what you do necessarily, so this broad analogy will sound cartoonish: sort of like questioning why a Windows text editor Github README didn't link to Microsoft.com and explain what Windows was.
There are those who might argue (and those like me who merely take a Tellarite stance https://memory-alpha.fandom.com/wiki/Tellarite) that they can define the audience for their paper by who would actually choose to use the model that they made or who might try to further refine it. Specifying the audience that way does a fair job of singling out people who don't know what the word "AI" means, and may or may not do a good job filtering out people who have never heard of "Whisper" in the context of AI before. Because who is trying to get a more efficient version of a model they've never before encountered or thought about?
In my mind, "Whisper is an artificial intelligence voice recognition framework for research and incorporation into other software" would accomplish these things neatly. It need not be as overly verbose as you suggest.
[0] - https://heywillow.io/
I'll be looking at this as soon as it is released tomorrow.
Separately, we have some Willow Inference Server improvements in the works that increase the speed of speech recognition on CPU by as much as 50% (depending on CPU supported instruction sets, etc).
Between that, the performance we already have, and this work it will be a dramatic improvement - even on CPU. I'm really looking forward to posting the benchmarks when all of this comes together!
So then we'll see whether we get the 6x model speedup on top of the stated 4x faster-whisper code speedup, at same/nearly same accuracy.
I would generally start with the assumption that if something is significantly faster the accuracy has to suffer a bit, but increasing model size and/or settings such as beam size to compensate should allow same accuracy and higher performance (just not all of the stated performance gain).
Ctranslate2 is a sleeper powerhouse project that enables a lot. They should be up front and center and get the credit they deserve.
It should be possible.
Yes, this is me praying to the benevolent HN gods that someone will pick this up and run with it. I don't have a GPU anywhere close to capable...
TLD a six year old ~$100 used GTX 1070 is roughly 5x faster than a Threadripper PRO 5955WX at a fraction of the cost and power.
[0] - https://heywillow.io/components/willow-inference-server/#ben...
Did you mean TIL?
Have you used Echo/Alexa and seen what people do with it?
"Alexa make an entry on my calendar for lunch with Guillermo, Brian, and Kyle next week Wednesday at noon at Giordano's on Ohio street in Chicago". From 10-15 feet away, often with all kinds of noise, echo, who knows what. A child mumbling french can get within range of an Echo device and do this (with varying degrees of success).
Yes a lot of that is handled on device in the audio frontend and elsewhere but it often still bleeds through and makes the fundamental speech recognition challenging. Not to mention bring your accent/voice/speech pattern.
That's firmly Whisper territory and doesn't even get into the flexible grammar, integrations, etc with entire other stacks.
Plus, many hundreds of millions of dollars and nearly a decade later Alexa still struggles with this.
However, wouldn't your described use-case be an activity that occurs after wakeword activation? Then handoff the rest of the audiostream to Whisper for transcription?
Yes, that's exactly what we do[0] (just like the commercial stuff).
Wake word and VAD are low-resource and even an ESP chip can handle that + stream. The ESP-BOX-3 is actually our main target device for voice hardware interface. It's the nearly infinite audio, speech, grammar, language, etc variability and complexity where you need the "big guns".
Another thing that seems to be getting lost on people - user expectations for voice interfaces are pretty high. If wake fails, a transcript is wrong, speech rec is slow, etc it's easier, faster, and far less frustrating to just take your phone out of your pocket. At that point why even have something poorly attempting to do voice?
[0] - https://heywillow.io/how-willow-works/#willow-inference-serv...
Do you see an eventual future where some notional "model-on-chip" would hard-wire something like whisper into a dedicated integrated low-power chip for these more demanding uses?
It’s certainly possible. However, consider the market dynamics.
Look at the Coral accelerator from Google. It’s $60. It has 6m TOPS.
Sounds great, until you dig just a little bit deeper.
It has 6-8mb of memory. A speech recognition model of sufficient quality for these tasks is measured in hundreds of megabytes. Non-starter.
Even with the might of Google behind it the price point, performance, memory, and therefore utility is quite limited for all but a few bespoke applications. Google also has a lot of experience with their TPUs from phones to datacenters so they reduced costs and benefited from shortcuts via that experience and scale.
Yet the capabilities and software ecosystem are pathetic, with even the official Python implementation not having a single commit for 18 months, being stuck on Python < 3.10.
A random $100 used Nvidia card has 8GB of VRAM, 6 TFLOPS, and over 200GB/s of memory bandwidth. CUDA is also hands down the most well supported software ecosystem. There isn’t anything in ML that doesn’t have tier 1 support for CUDA, and vice-versa. Even this ancient card fully supports CUDA 12, so its future proof well into a decade past release date.
If Google can’t pull off something targeting this market with reasonable availability, price points, and software support a new entrant in the field doesn’t stand a chance.
If someone tried to manufacture such a device between the low manufacturing/sales volume, additional memory, and software ecosystem it would likely come in at multiples of the cost of a used Nvidia GPU and even then it couldn’t remotely compete on software.
GPUs catch a lot of flack on power usage but here’s the thing: my GTX 1070 idles at 10 watts with all models loaded. It can do frigate, transcoding with plex/jellyfin, and Willow voice sessions in it’s sleep and still have 80% of the VRAM free for whatever else I want to throw on it down the line.
It’s very difficult to compete with. Not impossible, but a very special set of things would have to come together to stand a chance.
The only thing I can possibly think of is a Raspberry Pi variant with an NPU and unified memory, but even that ecosystem would have a lot of work ahead of it to match what Nvidia (a $1T company) has built over 15 years with CUDA.
If all you're doing is parsing "Alexa" out of the air... you're going to have a bad time because realistically, there is a contextual requirement. In AI applications, a proof-of-concept is great, but 99.9% accuracy is basically useless. Think if computer RAM is accurate 99.9% of the time... that's a broken tool.
If it takes 2 seconds to say "Alexa", that's 43,200 2-second chunks in a day, but if the listener is using a sliding window at 60hz, that's 5.2 million opportunities to screw up each day. 99.9% success of parsing a 2-second slice of audio is insufficient.
At some point, no matter how much training you do for ONLY the word "Alexa", you're going to start getting diminishing returns, in which the model to reach desired accuracy will start getting bigger and bigger for less and less improvement. Logical context analysis can easily bridge the gap for much larger gains.
From the paper, for short-form audio:
> the distil-large-v2 model achieves the lowest overall average WER of 10.1%. It is one percentage point higher than the large-v2 baseline, with 5.8 times faster inference speed and fewer than half the parameters.
Long-form is similar, except Distil-Whisper does slightly better than Whisper (fewer hallucinations, apparently).
10% WER seems awfully high, and doesn't match my experience with Whisper. Maybe my audio is nice and clean relative to their test set?
So I infer that these data sets are some deliberately difficult audio: call centre recordings with lots of background noise, phoneline quality audio etc. Maybe non-native speakers. If I only heard that sort of audio once I also might have an error rate of 10%.
I assume therefore that the 10% word error rate is on very difficult audio such as pilots speaking to Air Traffic Control (distorted or clipped microphones with significant background noise), which I personally find can be difficult to 100% understand even though I'm a native English speaker and when both pilots and air traffic control are native English speakers.
I built a tool in the mid-201Xs on an ASR engine with 20%+ WER, and even that was good enough for what we were trying to do.
At a larger level, say an exercise to transcribe a back catalogue of audio might need a $1000 GPU with the current model speeds to get the job done in a reasonable time. With models that run 6x faster it might be that a $200 GPU is sufficient. That could be quite a significant saving for a small company or charity etc.
That being said, even with this distillation there's still the aspect that Whisper isn't really designed for streaming. It's fairly simplistic and always deals with 30 second windows. I was expecting there to have been some sort of useful transform you could do to the model to avoid quite so much reprocessing per frame, but other than https://github.com/mit-han-lab/streaming-llm (which I'm not even sure directly helps) I haven't noticed anything out there.
Tho if you're looking for a standalone app then you can give this a go : https://github.com/alex-vt/WhisperInput and run it right on your phone :]
For now they both run regular openai whisper thus tiny.en but as you can see there's tons of impromvement potential with faster-whisper and now distill-whisper :D
Llama2 isn't as good as GPT-4 but it's a hell of a lot smarter at Q&A than Siri or Alexa or any of those things.
PSA: I will pay for such a thing if it's really good, privacy respecting, local-first, and preferably at least source available.
edit: in my 30minutes of playing with it, I didn't find a good sounding open-source text-to-speech model for the final stage of the pipeline.
If you haven't lived with speech recognition, it's not apparent that the command space for speech environment is significantly different from the command space for mouse and hands. In order to make the command space for speech work well, the speech recognition environment the application needs to present to the API with access to all functionality and data within the application.
What does Dragon do you for that Talon can't?
It looks quite useful if you need command and control. for general diction, not so much. To be fair, Dragon and other SR systems fail at speech driven editing.
99.9% of my dragon use is plane text dictation and when if the app is select-and -say enabled,edit and make corrections by voice. Speech commands as they are implemented are rarely useful mostly because there are too many of them to remember. Fortunately, my hands have recovered enough it is faster to type and mouse that it is to silently try to remember commands, construct what I want to say and then say it without stumbling or pausing then, undoing what was recognized, and trying again until I get it right.
one thing I'd like to do is difficult; have the same command give the same results in different contexts.
For example: say "make me root" and have the command recognize what machine on what network I'm connected to, sent the command "sudo su -<enter>" and then send the right password from my password manager without me having to type anything.
example: tail [forever] <service log> work the same no matter what distribution, of logging method (syslog vrs journald) and priv level. if I need a sudo before accessing log data put one in and give it the password if needed without any action om my part.
another should be possible is dictating into an app (like thunderbird) with text boxes and when focus is outside of the box, turn off recognition output. if you leave recognition output active outside of a text box then speech is typing hotkey commands from the letters in the words recognized.
disaster example: hotkey-stroking your way through your email, phone rings, you forgot to turn your mic off. at the end of the all, you recognize your out-of-context recognition has destroyed your mailbox is and you have no idea how to recover it.
The biggest thing is dragon is an out-of-the box solution. 15 mins (or less) install and I am dictating at high accuracy and a large vocabulary.
It's not even in the same galaxy.
Note that it's important to understand the realities here - short of something like an RTX 3090/4090 with LLama and every performance optimization available when it comes to responsiveness and accuracy competitive with commercial solutions this is a big challenge.
Even with the potential improvements of this work and optimizations like Ctranslate (used by our Willow Inference Server and faster-whisper) getting sub one second response times like Echo/Alexa more-or-less automatically calls for GPU even with every performance trick available. As I like to say when it comes to ML/AI/speech rec/speech synthesis if you bring a CPU to a GPU fight you're going to lose - and all of the commercial implementations are certainly using GPU/TPU on top of who knows what else they've come up with over the years and their immense spending.
To get an idea of how dramatic this is you can see the benchmarks with Ctranslate2/faster-whisper and our Willow Inference Server here[0].
Looking at those real-world numbers even at a claimed 6x performance improvement a mighty Threadripper PRO 5955WX can barely meet this goal with the models needed for voice assistant use cases under real world conditions (medium/large). Throw an LLM in the mix and you're sitting around waiting at least several seconds for a response, even with ridiculous hardware. On anything less than ridiculous hardware (including GPU) that becomes at least 10s of seconds very quickly.
At the fundamental level a seven year old $100 used GTX 1070 is approximately 5x faster than a monster CPU like the Threadripper PRO 5955WX - at a fraction of the cost and power. That's just for the first step (speech rec), to get something approaching Alexa-level you're in RTX 3090 territory because performance and VRAM.
Amazon has spent hundreds of millions of dollars (minimum) over the better part of a decade developing Echo/Alexa. The open source world has a long way to go to catch up.
[0] - https://heywillow.io/components/willow-inference-server/#ben...
You say "I really like Jason". But, your audience is developers:
prompt=json
"I really like Jason" => "I really like JSON"
The docs give some more detail about how to structure the prompts and has examples about what does and doesn't work, it's meant for this exact purpose.
It's effectively useless for helping the model transcribe new words in longer content. That also wouldn't be a long-term solution anyways... no one wants to compile a huge list of "words Whisper probably doesn't know" and have to pass those in every time the model is being used. Even if that worked, it would also distort the transcription, since you're not saying you know which words are in the actual speech, you're just passing in a list of words. So, you could end up influencing Whisper to choose the wrong words, giving priority to this list of random words being passed in.
I am similarly curious about how we can train Whisper models to learn new words over time, unless OpenAI plans to release updated models themselves.
- paper co-author