>> 1. Super Intelligent AGI is possible.
When?
When?
People are famously bad at predicting when right up until they have already done it.
“In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.
“In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.
“And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.”
Next question:
How?
How: keep adding more people and more technology and more connectivity to Earth. Simmer. This method has found superhuman strength (hydraulics), superhuman speed (wheeled vehicles), superhuman vision (telescopes/microscopes), superhuman calculating ability (calculators), superhuman memory (paper/computer storage), super-natural (in the literal sense) calorie sources (refined sugars and oils), and more. A human brain has an estimated 80 billion neurons, humanity is currently selling over a billion smartphones per year.
This may seem like a poor choice of method, but this method has been able to self-improve to develop precision machinery from nothing, control of electricity from nothing, large scale organization of groups of people from nothing, and more.
That makes for a crucial difference with the capability to develop superintelligence: we have no idea how to do it, and we've never created anything even remotely similar to it, yet. It's impossible to see how it might happen just by mixing up some components and stirring well.
Putting global collapse aside, another 10k years will pass, and another 10k after that. Is there good reason to think either that today is approximately as close to superintelligence as we can ever get (suspiciously arbitrary), or that the "next step" is so far out of reach that no lone genius, no thousand year focused group, no brute force, no studying of differing human intelligence, no unethical human experiments, can ever climb it? "We don't know how to do it today" doesn't convince me. For the last 10k years we have hardly stopped understanding new things and making new things, that's more convincing.
I'm not talking about global collapse, btw. I'm mainly expecting that scientific advances in the next couple hundred years will leap-frog today's unscientific research into artificial intelligence. I'm guessing that we will eventually understand intelligence and its relation to computation and that we will find out that today's ideas about artificial intelligence never made any sense, nor had any chance of leading to artificial intelligence, of any sort.
You see, I trust science. And it's obvious to me that the current dominant paradigm of AI research is not science. So I don't believe for a second that, that paradigm, can really achieve anything approaching intelligence, running on a digital computer. Because that sounds like a very hard thing, and the kind of very hard thing we can only do with science.