I’m confident that there’s no magic and I’ve spent years understanding “what the weights do.” You’re describing weights as magic and they are not.
Especially when you have billions of weights.
These models are finding general patterns that apply across all kinds of subjects. Patterns they aptly recognize and weave in all kinds of combinations. They are sensibly conversing on virtually every topic known to human kind. And can talk sensibly about any two topics, in conjunction. There is magic.
Not mystic magic, but we are going to learn a lot as we decode how their style of processing (after training) works. We don't have a good theory of how either LLM's or we "reason" in the intuitive sense. And yet they learn to do it. It will inspire improved and more efficient architectures.
I have also spent many years looking at weights!
Imagining we are really doing everything it does automatically including learning via algorithms we have only vague understandings of.
That is a strange thought. I could look at all my own brain's neurons, even with a heads up display showing all the activity clearly, and have no idea that it was me.