Deaf-blind authors would beg to differ.
But yes, a human brain is exposed to lots of other sensory input, and we know from other research that multi-modal models can learn shared representations that benefit from the knowledge of each domain.
In Transformer's favor, at least, they are far closer to tabula rasa than the human brain is and likely have to dedicate a lot of their training time to things that are otherwise "baked" into human brains. For example, humans come pre-packaged with V1 and V2 as part of their visual system, but CNNs and ViTs have to learn those filter packages from scratch.
I agree with you though. Human brains are able to take single instances of experiences and build a wealth of understanding from them in ways that even modern Transformer architectures are not yet able.