What often gets overlooked in these discussions is how much of the human brain is hardwired as a consequence of millions of years of evolution. Approximately 85% of human genes are used to encode the structure of the brain [0]. I find this particularly impressive when I consider how complex the rest of the body is. To relate this to LLMs, I'm tempted to think this is more like pre-training rather than straightforward model design.