>It could even be a fundamental rule of intelligent life, that one cannot create intelligence that surpasses its own.
Well that is easily disproved by the fact that people have children with higher IQ's than their own.
Well that is easily disproved by the fact that people have children with higher IQ's than their own.
Not to say we can't create machines that far surpass our abilities on a single or small set of axis.
First of all, it's not necessary for one person to build that super-intelligence all by themselves, or to understand it fully. It can be developed by a team, each of whom understands only a small part of the whole.
Secondly, it doesn't necessarily even require anybody to understand it. The way AI models are built today is by pressing "go" on a giant optimizer. We understand the inputs (data) and the optimizer machine (very expensive linear algebra) and the connective structure of the solution (transformer) but nobody fully understands the loss-minimizing solution that emerges from this process. We study these solutions empirically and are surprised by how they succeed and fail.
We may find we can keep improving the optimization machine, and tweaking the architecture, and eventually hit something with the capacity to grow beyond our own intelligence, and it's not a requirement that anyone understands how the resulting model works.
We also have many instances in nature and history of processes that follow this pattern, where one might expect to find a similar "law". Mammals can give birth to children that grow bigger than their parents. We can make metals puter than the crucible we melted them in. We can make machines more precise than the machines that made those parts. Evolution itself created human intelligence from the repeated application of very simple rules.
Yes, it seems likely to me.
It seems like the ultimate in hubris to assume we are capable of creating something we are not capable of ourselves.
So to create something that exceeds our capabilities is not a matter of hubris (as if physical laws cared about hubris anyway), it's an unambiguously ordinary occurrence.
SOTA models are superhuman in a narrow sense, in that they have solid background knowledge of pretty much any subject they've been trained on. That's great. But no, it doesn't turn your AI datacenter into "a country of geniuses".