It depends on your goal. To improve your listening skills for conversation, you'll have to engage in conversation and it will improve over time. It is normal to not be able to catch everything. Politely asking people to repeat themselves is perfectly fine.
One trick I use is to repeat what I thought I heard so the speaker can correct me. As opposed to just saying "what?" I would ask, "The cat was rolling weird?" This might get corrected to "The `Cart` was rolling". I find people will emphasis the word you're having trouble with.
I too had a decent TOEFL score(108/120), but I quickly realized that I could think of words in my head but not pronounce it. I still struggle with this and I get corrected all the time. Eg: My "bear" "beer" and "bare" sounds almost the same.
Reading books and articles helps with learning new words and keeping your vocabulary. Colleges generally have "writing centers" for help with formal writing. These people also help you with conversation and are generally happy to talk to you. I've also utilized places like #english on freenode.net sometimes.
Mimicry also helps. If you hear someone talk fluently, you should mentally commit that to memory. Some native speakers aren't good role model to mimic. So you want to actively make sure you're not copying wrong model. Most of this is initial work. Once you get to a thresh hold, you'll have easier time more or less.