Unless you're starting as a small child, you will never pick up words that do not occur frequently just from listening and interacting. Or it will take many more years than it ought to get to a given level.
Knowledge of a large number of these words is required to become competent, because, as a category, those words are coming at you all the time. Problem is, you don't know which one of the many thousands is coming your way next on any given day. The author of this article makes the same remark, basically:
> But there’s also an enormous amount of low-frequency words and syntax that even native speakers might encounter only once a year. Knowing any one of these “occasional” words or phrasings isn’t essential. But in every context — a book, an article or conversation — there will probably be several. They’re part of what gives native speech its richness.
If you don't know these words, you don't know a lot of what you're listening to. You might have this explained to you, and so then in that session you are okay with those words. However, without follow-up repetition, those words will soon evaporate.
I live in a highly multicultural country, Canada. Here you can encounter immigrants who have been here 25 years or more, whose English still sucks; and it's not always due to only associating with speakers of their native language. They use English everyday and interact with English speakers, all right. It's simply due to not mustering the academic wherewithal to study properly.