I guess it comes from the fact that when we are speaking IRL we have more senses available to us, while in written form we have just that single channel.
Anecdotally, I often find myself reading and notice a word I feel is familiar, but wouldn’t be able to give a clear definition. Majority of the time I will just skip over it and continue reading. I guess there is a level of redundancy there. After moving to the United States I had a manager who would use the word “Pollyanna” at least once a week in conversation and look me over a year of that before I finally looked it up to see what they were meaning.
From a cognitive & neurological point of view it’s pretty interesting that we can learn such a large written word vocabulary with these words that have such a low frequency of occurrences.