I'd rather see a Good Samaritan being talked through CPR or whatever by a dispatcher who's trained to give that advice over the phone, rather than having a hallucinating LLM tell them to do something deadly.
In some rural area of Africa they came across a car crash. Two people hop out and assist while a third drives off to notify someone to send emergency help.
An on device LLM might be very useful there depending on what it says…
I have shown up at scenes where people have googled what to do though and, you know what, it was super helpful.
If someone is dumb enough to perform a tracheotomy because an llm, google, or a passerby told them to. The issue isn’t any of those factors. That person is just so incredibly dumb as to be a danger to everyone around them.
People also do amazingly dumb things because a piece of software with a tone of authority told them to do it, even when they're not under duress. Look at the number of people who find themselves stranded or dead because they uncritically followed the directions of a navigation app, and who weren't in a panic state when they did it.