Imagine trying to learn to read English, and instead of seeing words, you see a bunch of letters and have to decipher each letter. I'd almost describe it as dyslexia for musical notation or something (which isn't a problem for me when reading text). I can recognize C,D,E,F,G pretty instantly because of their position relative to the bottom bar of the treble clef, but as soon as you start getting above that, I start having to count spaces relative to a known position, because everything internal looks like a big jumble. So G-B-E (on the treble clef) is easy because I see G as the base, and then the other two notes are space by two, but if it's just B-E without the context of the G, I have to stop and figure out exactly where those notes fall. And as soon as I get below C (on the treble clef), I start having to stop to count the separator lines.
But if I can hear what I'm trying to play, I can usually just jump to the correct note/chord. Maybe I'll have to stop and experiment to get the right chord periodically, but I don't have to stop and analyze the position of each dot on the score. To me, the greatest value of sheet music for me has always been in keeping place, so I associate the location of specific patterns on a specific page, and based on all the context, and know that I'm supposed to be at this point in the song that I've already taken the time to memorize beforehand. But I'm pretty much never paying attention to the actual notes on the notation at that point.