Likewise. I have five observations:
The more words I know, the more likely I can recognize them as they flow by....
By flow I mean listening to the whole sentence, paragraph, and not trying to hang on every phoneme: Between context and the human tendency to repeat, chances are you'll get the gist, especially if there are key words you can pick out.
I find the most important things to learn are "how do say", "what is the word for", "can you please repeat, but more slowly, please", and "sorry for mangling your language".
Learn the lilt of the language, how speakers hold their mouths, and how people use their bodies: The more you sound like you know the language, a) the less likely they are to switch to English, and b) the more pleased they will be with the effort you are making.
People are remarkably forgiving of poor grammar when its seems you are making a genuine and respectful effort.
(Native English, near-native and fluent French, joual, and a few other regionalisms, conversational Spanish that would likely come back pretty quickly, porto-conversational Turkish that might come back, market Arabic that might come back, really bad Greek, enough Polish and German once upon a time to at least order beer and food and get from A to B.)