If you speak with disfluencies, you probably didn't sufficiently rehearse your speech. If you didn't rehearse enough, you probably didn't put much effort into writing it either, so why should I put much effort into listening? It's the same principle as AI slop.
Many people can speak off the cuff fluently and confidently, avoiding "like", "um", and other filler words. And even if you're not speaking fluently, leaving silences as punctuation is more effective, IMO.
Many impressive speakers I've met actually cite Toastmasters! So their obsessive zeal actually does work.
More rehearsal does work too sometimes, but it does sometimes lead to speeches "sounding too rehearsed".
I don't think that's true, we usually just don't notice filler words in the same way we are surprised that people usually don't even talk in whole sentences, in contrast to written text or movies (which also use written text).