> There is just no reason to believe that we are born with some insanely big library of knowledge, and it sounds completely impossible. How would it be stored, and how would we even evolve it?

We do have that, ever felt fear of heights? That isn't learned, we are born with it. Same with fear of small moving objects like spiders or snakes.

Such things are learned/stored very different from memories, but its certainly there and we can see animals also have those. Like cats gets very scared of objects that are long and appear suddenly, like a cucumber, since their genetic instincts thinks its a snake.