Carmack: The code for AGI will be simple
youtube.com
youtube.com
Primitive now but in their time they were groundbreaking and incredibly difficult to make work on the primitive hardware of the time and he kept pushing the envelope for a long time. Carmack is a bit of a genius, but that doesn’t mean he’s right about AGI. My own take is that actual General Intelligence may rely on any physical phenomenon from the macro to quantum effects including aspects of reality we don’t know about yet. It’s really hard to simulate things when you don’t know they’re involved. We won’t know until we try so I guess we’ll see what happens.
This is generally how science works. 99% evolution in small incremental advancements, 1% revolution in a stroke of genius that opens up new areas of science.
Even humans (being barely intelligent, weak) are harmful to other humans (look at elites around the world: wars, coups, assassinatios, propaganda, etc).
Unless there is a fundamental barrier that handicaps intelligence at the human level (highly unlikely), AGI will eventually leave people far behind and destroy us in the process (like we destroy an ant hill, to build whatever we need for reasons incomprehensible to ants).
Anyway I think robots or other devices can't have will either way. For example if someone plants land mine and years later this land mine kills someone. It is not land mine's will. But of person who planted it. You can't sue land mine.
Sometimes, there are good reasons to just not build a thing. Takes a while for the idea to become apparent, but once you start thinking in terms of actually being responsible for second order and higher outcomes, it becomes much easier.
Take heart, though; the paperclip maximizer will probably just "want" us to watch ads and buy shit. A slow death, via titrated amounts of The Entertainment from Infinite Jest. At least it won't hurt too much.
But the path towards this goal is going to be anything but simple. To begin with it will need a multiple order of magnitude improvement in computational power. I primarily view this as a hardware problem.
Nothing in biology is simple. Nothing. Even the simplest things are orders of magnitude more complex than I ever could have imagined from my lofty platonic realm of numbers and functions.
Life is fractal. Every time you zoom in, there is just as much complexity as the previous zoom level.
Complex algorithms are not energy efficient and since the brain is highly energy efficient we must preclude such algorithms from consideration.
From what little I know, I am unwilling to make that assumption, but I do think that starting with a relatively simple algorithmic/mathematical biology-inspired take and slowly ratcheting up the complexity as needed is a reasonable way to proceed.
I posit the relevant details will be equivalent to a few tens of thousand lines of code since the brain is a highly (energy) efficient organ and there is no room for complex algorithms.
That is to say that while humans developed higher intelligence over a relatively short time, that was created from a novel combination of evolutionary abilities rather than a novel evolutionary invention.
Following JAX code is the essence of artificial general intelligence. It uses all simple tricks and techniques in tandem to achieve super-human meta-learning abilities.