So much this. I've managed to make learning fun for my three year old son. All too often we turn some daily scenario into a fun exercise and a well-meaning family member will exclaim "he can't possibly know that!". I assume they intend to shield him from the inevitable failure they believe I'm setting him up for by asking these questions, but he usually figures it out. And when he doesn't, he still gets a kick out of understanding it when we go through it together.
I repeatedly ask them to not make these comments. The more often he hears them say these things, the more liable he is to start believing them himself and say "I can't figure this out because I'm only x years old".
I believe that if kids were allowed to be challenged and excel at the things they show an interest in and predisposition to, the scholastic standard would be much higher. Instead of adults deciding what children of certain ages are "supposed to" be able to do and not to do.
My biggest fear at the moment is for his excitement at learning being crushed when he starts school.