And a random personal anecdote:
I had a really difficult time even at toastmasters dealing with my Uhms and Umms. Having someone count them somewhat works, but I decided to experiment by just watching my daily speech to be the same “quality” as public speech, and try to suppress the garbage words even during no-stress moments.
It took maybe half a year of practice, but as a result I started to speak slower and (probably :) clearer in the day to day life, and this during the public speaking too, with zero effort.
Edit-append: the most useful exercise is taking the projector, presenting in an empty room as if for the crowd, recording this “from a seat” and watching yourself. Extremely weird at first, but you very quickly see the areas to focus on. And since you must rehearse any talk anyway several times (at least that is the case for me), it is a zero-cost and high return activity.