The more literature I read about this, the more compelling I find this. In mammals, the larger the body the larger the brain. That's because you need all those neurons to control all those muscles. I recently read some papers that hypothesize the muscle atrophy we experience as we get older is partially the result of the neurons controlling those muscle-fibers dying [1]. As infants, our brains grow into our bodies by producing more neurons than we need and then killing off the neurons that aren't being used [2]. It makes sense to me that if we don't use our bodies in a variety of exercises, the lack of signal from the muscles to the brain could result in pruning those neurons as we get older.