Yes, they’d panic without training and practice. Because people panic, get frantic, or otherwise become emotional to the point of misunderstanding. Calm communication in serious, dangerous, or emergency situations is learned.
But other people who use radios, like the Army, don't speak in that drawl but also don't panic on the radio, and were doing it well before.
It's not the specific way of speaking that prevents panic. It's the training. It just happens that for a while pilots all wanted to sound like Chuck Yeager and then it just became the way new pilots talked because that's how all the other pilots they knew talked.
As a private pilot I learned what to say in the radio in case of emergency but they haven't tought us to be Yeager-calm. They told us to fly the airplane as the top priority, keep calm, think about the options and if the situation allows that, contact ATC and describe the problem with proper ATC prodecures. Of course it wouldn't help to freak out and yell to the radio. I never heard about Yeager before. I am sure he was great Air Force officer and perfect test pilot but I am not sure about this unique communication specialty either. Pilots are expected to be calm, well-trained, know the ACFT limitations and emergency procedures.