Hacker News Reader

Comment by gamegoblin | Hacker News Reader

I recently tried Whisper to transcribe our local Seattle Fire Department radio scanner -- unfortunately it was not reliable enough for my use case, e.g. "adult male hit by car" gets transcribed as "don't mail it by car".

I imagine future models will allow the user to input some context to disambiguate. Like if I could give it the audio along with the context "Seattle Fire Department and EMS radio traffic", it would bias towards the type of things you'd likely hear on such a channel.

View on HN

Have you tried the --initial_prompt CLI arg? For my use, I put a bunch of industry jargon and names that are commonly misspelled in there and that fixes 1/3 to 1/2 of the errors.

I was initially going to use Azure Cognitive Services and train it on a small amount of test data, after Whisper released for free I use Whisper + openai GPT-3 trained to fix the transcription errors by 1) taking a sample of transcripts by Whisper 2) fixing the errors and 3) fine-tuning GPT-3 by using the unfixed transcriptions as the prompt and the corrected transcripts as the result text.

Whisper with the --initial_prompt containing industry jargon plus training GPT-3 to fix the transcription errors should be nearly as accurate as using a custom-trained model in Azure Cognitive Services but at 5-10% of the cost. Biggest downside is the amount of labor to set that up, and the snail's pace of Whisper transcriptions.

There have been a lot of hacks to speed up whisper inference

Sweet! Do you have any links to resources on how to speed it up? I couldn't find any while searching Google or the Whisper discussion forums.

Not a hack per se but a complete reimplementation.

https://github.com/ggerganov/whisper.cpp

This is a C/C++ version of Whisper which uses the CPU. It's astoundingly fast. Maybe it won't work in your use case, but you should try!

Thanks for the tip, that did improve accuracy a lot.

The issue here is for most radio systems you end up with about 3 kHz of effective audio bandwidth (sampling). Most ASR/STT models are trained on at least 16 kHz audio.

Did you try a telephony oriented model like aspire or similar? They’re trained on sort-of 8 kHz audio and might work better.

I tried something similar for my SDR feeds and gave up because it’s just too challenging and niche - the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc all contribute to very challenges issues for an ML model.

> the sampling, the jargon, the 10 codes, the background noise, static on analog systems/drop outs on digital systems, rate of speech, etc

Is the reduced bandwidth really the most significant problem? Naively I'd think everything else you mentioned would matter a lot more, I'm curious how much you experimented with that specifically.

When it all comes together it's kind of a nightmare for an ASR model. There were plenty of times in reviewing the recordings and ASR output where I'd listen to the audio and have no idea what they said.

I'm not sure which contributes most but I know from my prior experiences with ASR for telephony even clean speech on pristine connections does much worse with models trained on 16 kHz being fed native 8 kHz audio that gets resampled.

I've done some early work with Whisper in the telephony domain (transcribing voicemails on Asterisk and Freeswitch) and the accuracy already seems to be quite a bit worse.

Could one train an interpolation layer (eg take a bunch of 16k audio, down sample to 8k, train 8k->16k upsampler)? Or better yet (but more expensive), take whisper, freeze it, and train the upsampler on whisper’s loss.

"I understand some of those words."

Hah, in all seriousness I'm more of a practitioner in this space. If this was something I absolutely needed to get done who knows where it would have went. For a little side hacking project once I encountered these issues I moved on - back in the day expectations were lower for telephony and the 8 kHz aspire models and kaldi were adequate to get that "real work" done.

Sure, that's called audio super resolution, there's a few papers/projects doing that. Haven't really seen models which are robust and have good generalization though.

We built a working version of this using Assembly.ai

I am no-longer involved in the project but you're welcome to contact the CTO if you're curious how it worked:

https://skyscraper.ai/

I (used to) use simonsaysai.com to generate subtitles and they had the functionality to input specialized vocabulary, so I suppose it's possible in some sense but I don't know how it would work with Whisper, something to ask on their Github if nobody else has yet I suppose.

But, for me, the English model works really well. Using the 'large' model works about perfectly for me, I can't think of anything I thought the large model got too badly wrong, is that the model you tried?

Yes, the problem is that the radio chatter is just very, very low quality, for a lot of words your brain just needs to know the context to fill in the gaps due to radio static and such. Even as a human some parts are unintelligible.

Yeah it's a hard case, Whisper with the large model is among the cutting edge in the business so if the static is bad and the quality is low there's not much you can do but wait for better AI, or fix whatever they get wrong by hand, but Whisper AI is on the cutting edge so you might have to wait for a bit lol

Was there a big difference in accuracy depending on which model you used?

Yes, large was by far the best, but still not accurate enough that I'd be willing to put it into a fully automated pipeline. It would have gotten it right probably 75% of the time. Anything other than the large model was far too bad to even think about using.

Whisper does pretty well, even with background music and things like that, I think you're working with a pretty weird subsection of recorded audio that won't work, for that edge case to work you'll very likely need to train your own model.

What was the performance, resource usage, etc of doing this with large? What's the speed like?

I'm still getting spun up on this but base delivers a pretty impressive 5-20x realtime on my RTX 3090. I haven't gotten around to trying the larger models and with only 24GB of VRAM I'm not sure what kind of success I'll have anyway...

In my case the goal was to actually generate tweets based on XYZ. As I've already said there were serious technical challenges so I abandoned the project but I was also a little concerned about the privacy, safety, etc issues of realtime or near-realtime reporting on public safety activity. I also streamed to broadcastify and it really seems like they insert artificial delay because of these concerns.

You can run the larger models just fine on a 3090. Large takes about 10G for transcribing English.

For a 1:17 file it takes:

6s for base.en, I think 2s to load the model based on the sound of my power supply.

33s for large, I think 11s of which is loading the model.

Varies a lot with how dense the audio file is, this was me giving a talk so not the fastest and quite clean audio.

While I saw near perfect or perfect performance on many things with smaller models, the largest really are better . I'll upload a gist in a but with Rap God passed through base.en and large.

edit -

Timings (explicitly marked as language en and task transcribe):

base.en => 23s

large => 2m50

Audio length 6m10

Results (nsfw, it's Rap God by Eminem): https://gist.github.com/IanCal/c3f9bcf91a79c43223ec59a56569c...

Base model does well, given that it's a rap. Large model just does incredibly, imo. Audio is very clear, but it does have music too.

> based on the sound of my power supply

Hah, I love that - "benchmark by fan speed".

Good to know - I've tried large and it works but in my case I'm using whisper-asr-webservice[0] which loads the configured model for each of the workers on startup. I have some prior experience with Gunicorn and other WSGI implementations so there's some playing around and benchmarking to be done on the configured number of workers as the GPU utilization of Whisper is a little spiky and whisper-asr-webservice does file format conversion on CPU via ffmpeg. Default was two workers, is now one but I've found as many as four with base can really improve overall utilization, response time, and scale (which certainly won't be possible with large).

OPs node+express implementation shells out to Whisper which gives more control (like runtime specification of model) but almost certainly has to end up slower and less efficient in the long run as the model is obviously loaded from scratch on each invocation. I'm front-ending whisper-asr-webservice with traefik so I could certainly do something like having two separate instances (one for base, another for large) at different URL paths but like I said I need to do some playing around with it. The other issue is if this is being made available to the public I doubt I'd be comfortable without front-ending the entire thing with Cloudflare (or similar) and Cloudflare (and others) have things like 100s timeouts for final HTTP response (Websockets could get around this).

Thanks for providing the Slim Shady examples, as a life-long hip hop enthusiast I'm not offended by the content in the slightest.

[0] - https://github.com/ahmetoner/whisper-asr-webservice

why did you want to transcribe it? What would you do with the output?

I wanted to make a twitter bot that posted whenever a pedestrian/cyclist got hit by a car.

I'd need to:

- Wait for a MED6 or AID response code to come across the live event stream

- Listen to the radio chatter to see if it was a pedestrian getting hit (use GPT3 on the transcription to determine if the text was about a ped/cyclist getting hit)

- Maybe also correlate to SPD logging a 'mvc with injuries' at the same location

Reply on news.ycombinator.com