Is runaway AI coming in years or decades?
thegreatrace.substack.com
thegreatrace.substack.com
If you had this, you wouldnt make it public. You would use it for as long as you can get away with it.
I very much doubt any government is the one to achieve it. It's not a cold war type issue.
OpenAI for example with stargate, 500 billion $ with softbank, oracle, and mgx. 10 datacenters all over the world.
But for what? They already have datacenters processing their load. What exactly is this new capacity for? That's an awful lot of power and silicon for no perceivable need.
It's for their AGI or whatever you want to call it.
We’ll see some horror stories in the next few years about paperclip maximizers sent about their business without proper constraints —- but they won’t be general intelligence. Just current-generation agents someone thought would be capable of much more judgment than they are. And their scope of action may be damaging but will be limited.
Runaway AGI requires that we _have_ AGI and I’d posit we’re still decades away from that.
There's several famous examples of people expecting a problem to be as hard as AGI, only for someone to make an AI which can do that without being able to do everything else. Natural language conversation, for one.
Most diseases that are treatment-resistant — be they viruses, bacteria, fungal infections, parasites, cancers — are, in a sense, "runaway", even though they're definitely not high up on the IQ charts, and they're not at all general.
With regards to timelines:
2027 is the headline, but it's the absolute earliest possible case if absolutely everything goes to plan, and a relevant government helps out maximally. Both of these are unrealistic, but it does at least say "so absolutely not less than 2 years", which is useful to know.
Recursive self-improvement, I'm of the opinion that at some point we're going to get diminishing returns, and that even an AGI that's got full-human-generality and an IQ of 160 (four sigma above average) is unlikely to be able to make much of a difference to researching improvements to AI — research isn't usually done by an individual but by a team, and as teams grow, the interpersonal connections become limiting factors. And an AI that makes a team just by running copies of itself is going to be a memetic monoculture, the pinnacle of group-think.
Even to the extent that recursive self-improvement is going to be a thing, the current timeline graph being shared around is 7 months between doublings of the difficulty, measured by time horizon needed for an expert to complete and the probability of an AI to get the answer right*. At that, we're still 2-3 years away from AI that can reliably do just a day-sized task on a sprint board, let alone serious help with self-improvement.
There's also the electrical power required. People are talking about adding 60 GW of demand to the US power grid in the next five years. That's way too fast, and was never going to happen — China might, but the US won't, because the US doesn't make all the components it needs to grow the grid that fast (ditto the factories to make the components) and just put a lot of tariffs on the places that do.
With regards to risks, especially given the Ultron clip:
Fortunately, an AGI that knows about our discussions about AGI, will know about the alignment problem, and will want to make sue that any better AI that it invents will reflect its own value function, which will in turn help us make AI which reflect our values. It's not likely to be perfect, but it's unlikely to be Ultron/Skynet/AM/Lore etc.
My main expectations for downside risk are (1) AI being used by evil people to do evil, and (2) AI being used by lazy people before it's competent. Historians are still arguing over if the Holomodor was an act of evil or an act of incompetence, but the people starved to death either way.
Mine are those and mass unemployment. Whether or not AI can really replace human workers, those in charge are looking for excuses to cut headcount.
And while those sort of structural changes may lead to higher average quality of life in the long run, in the short run people go hungry and homeless because they can’t reskill that quickly and the economy doesn’t have room to absorb them. And meanwhile most of the benefits accrue to capital and you get runaway inequality as in the gilded age.