Some of it is surely teething problems, but unless there is a robust check upon implementation it might just add another layer of inefficient new public management make-work to the system.
https://sverigesradio.se/artikel/ai-journaler-i-sjukvarden-k...
Anyway, it creates the very problem you mentioned but just replace "road" with "outside the cockpit".
What do you mean by this? Not having an IFR rating does not mean you're not allowed to use the navigation aids or the plane's autopilot.
You can use them all you like. You just can't fly in conditions where you have to use them.
The article does not suggest that doctors should blindly trust the SOAP note created by the tool in question.
But that's what will inevitably happen at some point, when they get to the point of only rarely making big dangerous mistakes.
Are you intentionally rubbing FUD on that or am I mis-reading you? I don’t think we need to wait to rely on technologies until they’ve achieved perfection - just when their mistakes are less frequent, less dangerous, or more predictable than human mistakes for the same task.
As a concrete example: https://www.cbsnews.com/news/pilots-fall-asleep-mid-flight-1...
There should be requirements for any AI tool provider in the medical space to go through something like an IRB (https://en.wikipedia.org/wiki/Institutional_review_board) given they're fundamentally conducting medical experimentation on patients, and patients should have to consent to its use.
It is important to remind people that technology of any sort can be error prone and that human oversight should be relied on for any automated process, LLM based or not!
I work in the legal industry and every lawyer is aware of the guy who used ChatGPT to spit out non-existing case law!
https://text.npr.org/2023/12/30/1222273745/michael-cohen-ai-...
So the same as doctors making occasional big dangerous mistakes that cause lives. Seems like it would be a win then as it takes some mental load off of doctors so they can focus on where they should, on the patients and not on note taking.
Will it be? There are already unanswered questions on who's liable if Tesla's FSD runs you into someone.
If an auto-form filler is not working correctly the doctor can also recognize the issue and also be in a position to do something about it, namely, fix the error before they submit the form.
That is to say that there's a world of a difference between a pilot flying a plane and a doctor filling out a form.
To save people looking it up, that one-char difference changes "abdominal pain" into "cock pain".
Wow.
That is the main problem with the AI: it is close enough, but never there.
At the beginning they will be worse than humans and cause deaths that humans would have prevented while at the same time probably saving lives where a human would make a mistake.
But not far down the road they'll become much better than humans even if they do occasionally make a mistake and cause a death that a human wouldn't' have they'll save far more lives due to them not making the mistakes that humans do.
While I think you're correct, there is no proof this will ever be achieved.
It very well may not be possible along our current path. It may take a 100 years, 1000 to get there.
And yes it could take only 20 more. But to state this as a certainty?
No.
That's what we're talking about here. IMHO computers are already better at that, what possibly makes you think this won't happen?
This seems very similar to self driving cars.
And continued discussing self driving cars.
Regardless, what I said stands.
I've been transcribing A LOT of SR (Swedish Radio) shows as part of https://nyheter.sh/, and Whisper (self-hosted) has been very accurate.
Whisper + an LLM can recover some of the gaps by filling in contextually plausible bits, but then it's not a transcript and may contain hallucinations.
There are alternatives that share Whisper internal states with an LLM to improve ASR, as well as approaches that sample N-best hypotheses from Whisper and fine-tune an LLM to distill the hypotheses into a single output. Haven't looked too much into these yet given how expensive each component is to run independently.
I can't speak to how it performs outside of production quality audio, but in the hundreds of hours of subtitles that I've generated I don't think I've seen a single error.
Traditional ASR systems struggle when English (or any language) is spoken with a heavy accent, often confusing it with another language. Whisper is also affected by this issue, as you noted.
The root of this problem lies in how language detection typically works. It relies on analyzing audio via MFCC (Mel Frequency Cepstrum Coefficient), a method inspired by human auditory perception.
MFCC is a part of the "psychoacoustic" field, focusing on how we perceive sound. It emphasizes lower frequencies and uses techniques like normalized Fourier decomposition to convert audio into a frequency spectrum.
However, this approach has a limitation: it's based purely on acoustics. So, if you speak English with a strong accent, the system may not understand the content but instead judge based on your prosody (rhythm, stress, intonation).
With the team at Gladia, we've developed a hybrid approach that combines psycho-acoustic features with content understanding for dynamic language detection.
In simple terms, our system doesn't just listen to how you speak but also understands what you're saying. This dual approach allows for efficient code-switching and doesn't let strong accents fall through the cracks. The system is based on optimized Whisper, among other models.
In the end, we managed to solve 99% of edge cases involving strong accents, despite the initial Whisper bias there. We've also worked a lot on hallucinations as a separate problem, which resulted in our proprietary model called Whisper-Zero.
If you want to give it a try, there's a free tier available. I'm happy to bounce around ideas on this topic any time; it's super fascinating to me.
Humans also have difficulty with heavy accents, no?
My understanding on whisper is that it is using a model trained on different accents, specifically from LibriVox. The quality would depend on the specific model selected.
The MFCC or other acoustic analysis is to detect the specific phonemes of speech. This is well understood (e.g. the first 3 formants corresponding to the vowels and their relative positions between speakers), and the inverse is used for a lot of the modern TTS engines where the MFCC is predicted and the waveform reconstructed from that (see e.g. https://pytorch.org/audio/stable/transforms.html).
Some words can change depending on adjacency to other words, or other speech phenomena (like H dropping) can alter the pronunciation of words. Then you have various homophones in different accents. All of these make it hard to go from the audio/phonetic representation to transcriptions.
This is in part why a relatively recent approach is to train the models on the actual spoken text and not the phonetics, so it can learn to disambiguate these issues. Note that this is not perfect, as TTS models like coqui-ai will often mispronounce words in different contexts as the result of a lack of training data or similar issues.
I'm wondering if it makes sense to train the models with the audio, phonetic transcriptions, and the text and score it on both phonetic and text accuracy. The idea being that it can learn what the different phonemes sound like and how they vary between speakers to try and stabilise the transcriptions and TTS output. The model would then be able to refer to both the audio and the phonemes when making the transcriptions, or for TTS to predict the phonemes then the phonemes as an additional input with the text to generate the audio -- i.e. it can use the text to infer things like prosody.
Any recommended ones you've looked at?
The one time I used AI meeting notes, some important details were wrong. And beyond that, the notes were just terrible. A literal transcription can be useful. A summary of the substance of the meeting can be useful. This was neither. A human would know that a tangent talking about the weather is not important, but AI notes are just as likely to fill the document with "Chris mentioned it had rained yesterday but he was hoping to cook hamburgers when the sun comes out. Alice and Bob expressed opinions about side dishes, with the consensus being that fries are more appropriate than potato salad." as it was to miss a nuanced point that a human would have recorded because they understood the purpose of the meeting. And then it'd give me an action item to buy corn.
Just gonna adjust the temperature of your baby here, ok, now he should grow up just fine.