Hacker News Reader

This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.

Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.

Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.

basically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.

Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.

You 100% can finetune or adapt/build on top of models, and specialize them or extend their capabilities. That’s literally what post training is.

The problem is that “finetuning” was a 2023 AI FOTM associated with products/demos that were almost exclusively using it for LLM character role-play/output style purposes (ie not in actual systems where they served a more functional role).

This made people think you could train models without replay/real evals by yoloing it with SFT (this is partially an artifact of that era being much heavier on autoregressive training and not so much evals). You really can finetune and get results but you have to treat it like a small ML training run, with real evals, and more intentionality than just “more examples”.

You can find pretrained and -instruct models on huggingface that clearly demonstrate what specialization/staged training runs do.

I’d be very wary of conflating finetuning with specialization/extending a model’s capabilities in general.

Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.

Turns out the world is made of simple, specialist processes, not generalists trying to achieve them. Adaptability may be of great benefit in evolutionary terms or for a walking anthropoid, but the majority of biology, chemistry, and mathematics rely upon specialist process for good reason. See also the old trope about robotics: that's what you call it before it works, otherwise it'd be a dishwasher.

The upshot is: use a generalist to create a simple solution once, and scale that. Don't deploy the generalist at scale, that's a waste of resources and an inefficient solution.

> The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages.

I think this claim is acceptable in context, but taken alone, this needs to be qualified. Some aspects of language are universal, like abstract information structure and other pragmatics, but no model is going to speak rural Khmer dialects well anytime soon because not enough of it has been digitized, fundamentally speaking, hence qualifying what it means to "be good at all languages".

<< all human languages have the same underlying pattern structure,

Chomsky would like to have a word. Your statement is true only at a surface leve l( they have nouns, verbs and some limits), but it breaks down the moment you start to inspect it closely:

- sign language ( I am not being petty; you put all languages ) - it is almost nothing like the underlying structure of other spoken languages primarily because it does not carry its restrictions - English vs Polish example - word can carry grammar or not; word order can carry meaning or not

Those are just two examples, but both clearly show that little about human languages is actually the same. It is kinda like the history thing. It rhymes.

I am addressing this part as other posters noted issues with other parts.