Yes, that's the thing. There's literally millions of people in Africa who get little or no animal protein, and they grow up just fine. The point is that study after study shows that with just a tiny portion of animal protein, they end up a whole lot smarter.
I'm not trying to suggest you aren't yourself a smart person, just that simply because you can't see the ill effects your actions cause doesn't mean they don't exist.