Who can't remember the smell of more than a few dozen objects?
Look at our memory for faces. It isn't all about eyesight. Our brains have specific circuitry for faces. Dogs have the same for smell. Sensitivity is one thing, how the brain is wired is another.
I rememeber:
* how English Cadbury's smells
* how El Rey Cordoba milk chocolate (from Venezuela) smells
* how Hershey's smells
* how hot chocolate made with Swiss Miss mix smells
* how Toblerone smells
We're wired to be able to pull up those memories and imagine a smell, but not to be able to identify what we're smelling when it's right in front of us.
I think you are speculating cause.
Culturally we tend not to communicate much about smells - it could just as easily be that most of us have the ability but it is never trained because it isn't much use to us.
Anecdotally I know people with a wide range of different olfactory abilities.
It is interesting to try and think of how we could test for the ability. Perhaps not testing for my pencil, but something else - a child - a dog - shoes - urine.