Cleaning Up Speech Recognition with GPT | Hacker News Reader

Cleaning Up Speech Recognition with GPT

blog.nawaz.org

28 pointsBeetleB18 comments

"I’ll probably write an elisp function to send the output to GPT and insert the results into the buffer"

I suggest taking a look at my LLM command-line tool. It's great for cobbling together these kinds of things, because you can use whatever shell integration your environment has to pipe things to it.

    cat bad-dictation.txt | \
    llm -m gpt-4-turbo --system '
    You are going to correct for text that
    has been produced by voice recognition
    software. Rewrite any text provided to
    you. The text will not have punctuation,
    so please add where needed. If the text
    has the word period, then insert a
    period, and do not insert another one
    just before or after. If the text has
    “new line” or “newline”, insert a newline
    character and start a new paragraph. If
    the text says “comma”, insert a comma. If
    the text has a number spelled out,
    replace it with the actual number. So
    “five thousand” becomes “5000”. Also, if
    something seems off in the text, it was
    probably due to a misrecognized word.
    Please correct for it.' > out.txt

Should be easy to call that from elisp, and you can then install plugins to have it talk to other models like Claude or Llama 3: https://llm.datasette.io/en/stable/plugins/directory.html

Thanks a lot for llm, Simon. I've started to integrate it more and more in what I do as an abstraction layer over llm apis.

By the way, do you know of any models that support streaming input tokens to reduce latency to output tokens?

I don't think I've seen one that does that.

Groq is pretty amazing to play with to see what fast models are like (llm-groq plugin).

Thanks. The problem with groq is time to first token (being in a queue, presumably).

I wrote it in December, when I was none the wiser. I've already got an elisp function (using whisper now for the backend - which has some cons to it).

In fact, it's fairly common for me to use the combination of Whisper and GPT to compose emails, etc. I should do a followup post on my setup.

Interestingly, Simon, I also submitted another story[1] where I refer to your shot-scraper. You're the gift that keeps on giving (especially given that my first programming job was with Django in 2008!)

[1] https://news.ycombinator.com/item?id=40174802

BTW I’ve heard from researchers from a big GAFAM (can’t name) that they actually use gpt to do this on the ground truth and spot that correction and do a second pass of human labelling on the ground truth to have better dataset.

Thanks for sharing this!

Direct link to the repo for those curious - https://github.com/simonw/llm

If you prefer an existing webapp over an elisp function, https://huggingface.co/spaces/ndurner/oai_chat. Choose Whisper as the model, upload your 25 MB chunks, hit Send, choose GPT-4 Turbo, ask it to clean up, hit Send. Then, hit the Download button (hidden away on the very bottom).

Helpful fact: Whisper works on 16 KHz sampling behind the scenes, so can make your recording smaller by downsampling to 22 KHz, mono. AAC is supported, and commenters to the web say Whisper is pretty robust so it doesn‘t have do be hi-fi - just so that you can make a split at the beginning of the QA session perhaps, if you can‘t fit it into one 25 MB chunk right away.

I've tried using LLMs to restore and clean up raw unpunctuated transcripts, but they tend to hallucinate new words. And chunking is an issue since long transcripts need to be split according to the LLM input context size. But where do we split if we don't have the punctuation yet?

In Scribe I chose instead to restore punctuation marks using a token-classifier (here a DistilBert running in your browser) https://www.appblit.com/scribe

Laurent

I take a lot of long voice notes thinking out loud during my walks. I use this prompt when I give the Whisper output to GPT-4. I've found it to be pretty reliable:

> Punctuate the following transcript of a voice note I took. Insert periods, commas, and paragraph breaks where appropriate. Remove filler words such as 'right?' 'you know', and 'uh'. But do retain my original wording! Do not paraphrase my sentences beyond recognition: this is not a rewriting task! The transcript now follows:

Thanks! I'll give it a try. My current prompt is much longer (lists many things Whisper misrecognizes so GPT can replace them).

Did not think of telling it to remove filler words, but I find it usually removes them anyway.

Use whisper (or whisperx)

Whisperx is great for English, and there's minimal cleanup to do at the end. (Diarization makes it take much longer, and sometimes you get a bunch of repeated phrases, but transcript-only is good.)

But for other languages, especially those with a variety of dialects (like Arabic), the output can be really hard to read. Interestingly, gpt-3.5-turbo can turn a garbled Arabic call transcript into a decent English summary. I guess this is due to the amount of redundancy/repetition in phone calls.

I wonder whether anyone knows of something better than Whisper Large v2/v3 for Arabic ASR?

I had a look recently, and what I found was:

- Among published multilingual models (i.e. ignoring English-only models), it seems that Whisper Large has the lowest (best) overall WER (word error rate).

- Although Facebook published a multilingual model (mms-1b-all) that outperforms Whisper on many less common languages, its Arabic WER on standard benchmarks is much worse than that of Whisper, so I don't think it's worth trying on my data.

- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.

- Googling turned up some attempts to finetune mms-1b-all with Arabic, but nothing I found included any WER data, so I assume these attempts didn't work out.

That’s a bit problem with Arabic especially because of dialects and the lack of good datasets (when I say I mean robust - including noisy dirty audios).Would you mind testing gladia.io and give feedbacks ?

Before I try it out, would you mind sharing any info about your models' WER on Arabic, compared with Whisper Large V2?

“- There's a paper from ~6 months ago by some researchers who claimed to get slightly better performance than Whisper vanilla, but their code is not public, and I can't find any blog posts or articles talking about using their work.”

Have you got a link to this paper?

https://arxiv.org/abs/2310.11069

As I mention in another comment, I actually do now use Whisper. In fact, I just spent the last few hours using Whisper and GPT to write some drafts.

Whisper has a very serious downside: It often ignores quite a bit of relevant audio, and I dealt with this many times in the last few hours, to the point where I'm considering finding an alternative.

See this report, for example:

https://community.openai.com/t/whisper-leaves-out-chunks-of-...

Reply on news.ycombinator.com