The great thing about gravity is you can plug on a couple masses to an equation and know exactly what will happen. On the other hand, we have lots of short hand for what the brain is doing (eg, "memory") and lots of just so stories about how those things might work (eg, "like a stick of RAM"), but those stories rarely give a full picture of what's going on it help predict how to improve things - like remember, or cure Alzheimer's.
For many complex systems, simple observation is a problem. With a brain, you can't observe things in fine detail without killing the host. So we can't see how local phenomena interact to create complex emergent behavior - observing the local means suppressing the global. And while we have big global observations from MRIs, and tiny local observations from cutting out individual neurons and applying electricity, there's a huge middle ground in between.
Another problem is that global emergent phenomena can change the way the local systems behave... We tend to imagine knowledge progressing from the tiny systems or to the big systems, but we see feedback in the other direction too, for example through hormone release. Which, again, is difficult to observe when isolating tiny parts of a system.
Think of it like this. You've got a dog. Your dog is very smart. She knows her name, she does tricks, she has a pretty apparent understanding several spoken words, tonal inflections, and body language, and can even catch a frisbee out of the air when you throw one. However, no matter what you do, no matter how hard you or she tries, your dog will never understand calculus. A dog is simply not capable of understanding what calculus is or what it is for or how it describes relationships of objects in Newtonian physics. There is a fundamental physiological limit to her intelligence, and topics like calculus are beyond it.
Humans are much more capable than dogs. However, humans are still physical beings. As such, they have physical limits to brain size, and physical limits to brain activity. We also have a limited life span, so if something exists which might be comprehensible to a 200 year old human, we may never know it. It is most probable that there are concepts, topics, and systems that are simply too complex for human intelligence to understand even if we had access to the equivalent of a Calculus For Dummies.
Worse, human brains are limited to their senses, and scientific understanding is contingent on both observation and repetition. A system can only operate on the data it has available to it. If something exists which either cannot be perceived or cannot be repeated, then we have essentially no capability of even knowing that we don't understand it. The fact that some of the most complex systems have been understood though abstraction, encapsulation, and modeling doesn't mean that all possible systems will behave that way or can be understood that way or, frankly, even that the knowledge we gain about systems that way can truly be called "understanding." How often do we say, "It wasn't until I experienced it myself that I understood"?
As far as AI, well, we don't even know if they're capable of consciousness or sentience, let alone sapience. They may just be arbitrarily complex systems good enough to simulate human behavior and fool a Turing test. No, passing a Turing test doesn't mean you're conscious or sentient; it means you're not definitely non-sentient or inanimate. And if the basis of everything an AI knows is what they learned from us using algorithms we decided to give them, what does that mean they won't be capable of just because we're the ones that taught them?
TLDR: The universe is not obligated to function in a manner comprehensible to human intelligence.
"...it may be that humans are simply not intelligent enough to discover or to understand the final theory. It is possible to train dogs to do all sorts of clever things, but I doubt that anyone will ever train a dog to use quantum mechanics to calculate atomic energy levels."
The impossibility of dogs doing Calculus is an equally nice example, but I think we could push the point further. Calculus is not only hard and unnatural for dogs; it is hard and unnatural for humans as well. There were anatomically modern humans equipped with culture and language for something like 30,000 years before even one of them understood calculus. It's truly extraordinary that this thing which was at the absolute limits of the human intellect in the 17th century is now taught routinely in high schools and colleges across the world, and used for such mundane purposes as building bridges or stock trading.
Those inclined to optimism could read in this a story about how humans are not like dogs, because culture. (i.e.: "culture allows us to consolidate and simply the understanding of previous generations so future generations can build on it towards almost limitless horizons", etc.). But I'm a pessimist: calculus is still damn hard for nearly all human beings (just ask your average college student). Heck, I found it pretty tough to learn and I ended up getting a masters in math. And - to get back to the article - I'm not convinced that current topics at the edge of human understanding today (string theory, or whatever) are ever going to get 'mundane'. It takes many years of special training and extraordinarily single-minded dedication to even get to the current frontier of knowledge in these areas. At some point, as the article says, perhaps the math just gets too hard.
Fortunately, just because the universe might not be comprehensible by humans (we never get to a successful theory of quantum gravity, or whatever) doesn't mean science itself will have a end. There's plenty to do in other fields.
In other words, do we expect a dog to ever understand the final theory of physics? Surely not. Do we expect a 2 year old human to ever understand the final theory of physics? Still, presumably not.
And even if that argument isn't good enough; the dog has no tools to escape its constrains; humans do; we are messing with our brains constantly looking for a way to make it better; may it be by genetic therapy (DNA mods), chemically induced (drugs) or hardware implants (artificial neurons).
An analogy is just a described mental model. A "thing that is thrown" is something a dog can understand. I throw a ball or a bone or an orange mouse toy, it makes no difference to the dog...unless I "fake throw" or perform a "magic trick". I think the dog is likely to understand some part of calculus, given it has a sufficient number of neurons and was to use them optimally. The study into injecting information into a monkey's brain is particularly tantalizing toward that end.
A mental model is the primary tool necessary to escape constraints e.g. the ingenuity of Crows. Without external pressure or evolutionary pressure, humans haven't observed most animals get measurably smarter.
The set of things humans are in principle capable of understanding at least slightly are covered by the analogy about dogs. Try to imagine the relative complexity of concepts, for example how much more complex the concept of calculus is than the trick to sit when given a verbal command. Now imagine some concept that is equally more complex compared to the most advanced human math/physics as calculus is to the sit-trick. There is some scale of complexity where analogies break down, and even though they convey some concept that is familiar to the audience, its connection to the actual matter is so vague and strenuous that you haven't actually explained anything. There may be facts about the universe that are completely counter-intuitive, and the understanding of which depend on a regression of trillions of other counter-intuitive facts and processes.
We already see computer assisted mathematical proofs heading in the direction where there are simply too many steps for a human to understand. And thats merely us scratching the surface of the tip of the iceberg. At some point, I believe, computers will generate new maths such that not only is the proof incomprehensible, but the result too. There is no reason to believe that the universe is simple enough that humans can understand everything.
> we are messing with our brains constantly looking for a way to make it better; may it be by genetic therapy (DNA mods), chemically induced (drugs) or hardware implants (artificial neurons).
What makes you so sure humans can bootstrap their way into ever higher levels of intelligence, ad infinitum? Sure, it's possible to take that view, but I would say it's incredibly optimistic. And not one of those ideas you mention has yet made humans any better at doing science (well, maybe 'drugs' - scientists do love their coffee).
True, except it's exceedingly unlikely to be so. The SKI calculus, that can represent any effectively enumerable function, no matter how complex, is itself incredibly simple.
Complexity trivially emerges from simplicity, so the odds that most of the complexity we see around us is irreducible seems very low.
Which is why we'll never learn about atoms or X-rays.
But that just switches the limit to the limit of our computing power. Which is a pretty strict limit. For example, we can't construct computer models to predict the properties of all the elements in the periodic table just from knowledge of the underlying laws of electrons, protons, and neutrons.
Hmm, or can we? This gave me the idea of trying to extract "emergent laws" using machine learning.
There should be a way to build an AI that tries to find rules that approximate behaviour of complex systems. I'm sure there's lots of work on this already.
This is not what I was talking about. I was talking about a case where we know the underlying laws (the electromagnetic and strong interactions), but the computer models that we would construct from those laws would take more time than the age of the universe to run on our current computers and give us useful output.
Using computers to try and infer underlying laws, not currently known to us, from data is a different thing, and I believe there is indeed ongoing work in this area.
Yet. And the upper bounds on available computing are not actually known. Well, not strictly true, as arguments like the Bekenstein Bound suggest that information density that gets too high will collapse into a black hole.
But typically, computing properties with precision involves exploiting symmetries and other properties to eliminate impossible cases from consideration and enjoy a shortcut to the answer.
Just think of your average decently sized code base. There's enough complexity there such that any single human can't understand it at a given time, save for a very high-level explanation.
Does this mean that we don't understand the code base? I don't know if that's the right way to think about it.
Why do you think emergent properties are not the sum of their parts?
If I may simplify, there are generally two classes of emergence, metaphysical and epistemic. The former claims that systems are more than the sum of their parts, that some magic happens when some combination of things are brought together. Few people in science or philosophy takes this position seriously.
Epistemic emergence is more about what approach to use to learn about a system. Science is typically reductionist, and it's very successful. However, some properties would be difficult to discover this way because they are only observed in complex interactions between multiple entwined subsystems.
For a simplistic example, consider a scenario where you know nothing about physics, but are given unlimited time and unlimited tools to separately test the properties of oxygen molecules, and hydrogen molecules. But, you can never test them together, only separately.
Given these constraints, it seems virtually impossible to infer the property that H2O is transparent to visual light. But this property is easy to see when studying them together.
Epistemic emergence would hold that we maybe couldn't learn everything about hydrogen and oxygen by studying them separately, or at the very least, doing so isn't always the most effective means to learn about them. But ultimately, the properties revealed of H and O belong to them, and aren't metaphysical.