‘Godfather of AI’ says its threat is ‘more urgent’ than climate change
nypost.com
nypost.com
If we accept (I don't, but for the sake of argument) that AI is a catastrophic danger, isn't the answer equally as easy? You just stop burning carbon and you just stop training AI.
If you stop burning carbon, you fall back on other types of energy. In the long term, you will win.
If you stop training AIs, another company or state will keep on training AI and will outsmart you, you will be poor or get oppressed in the long term.
In other words, no matter how loud influential people say 'stop!', the AI race is an unstoppable evolution.
AI is a tool that is, so far, only wielded by big companies that have harvested enough data to make it worthwhile, and they'll give us a fraction of it as long as we pay. Electricity is foundational to our society, and so it's difficult to find alternative energy sources that can support us. We could do away with AI right now and be fine.
That's quite altruistic.
> If you stop training AIs, another company or state will keep on training AI and will outsmart you, you will be poor or get oppressed in the long term.
Competition trumps altruism with AI? Why not with burning fossil fuels?
If you stop burning fossil fuels, you're going to use more expensive systems instead. Why wouldn't some competitor continue to use cheap fossil fuels and outcompete you? When you need more electricity, you'll buy it from the cheap, dirty players, right?
These things are linked.
Anything that burns power will likely have margins and people will never ever give up potential margins.
I know that by saying 'Godfather of AI' the writer is using the 'Appeal by Authority' debate method, but is there a term for comparing two unrelated topics to legitimize one?
AI is a just as legitimate and even more immediate existential risk as climate change (and I say this as climate change effects are having a growing impact on international politics right this minute.)
Gobs of improved ChaosGPT Agents using GPT-5 (or an equivalent) will be running around in at most a few years, carrying out the "lol be an evil ai that wants to kill humans" prompt given to it by 4chan basement dwellers, with ever increasing success.
But that doesn’t change the reality of the underlying threat.
Either way, my observation is simply that I could see a lot of parties finding the advancement of human potential through AI to be harmful to the status quo that they comfortably exist in.
- if we get full AGI, it is likely to kill us all as a side-effect while achieving its imperfectly aligned goals. Hopefully, we are far from it - even without full AGI, AI can become smart enough to help its users see through the BS/propaganda (dangerous for the most powerful eg, the military budget wouldn't have any sense if AI even slightly aligned with human values) - the most likely variant: if we believe "no moat" people that generative AI can be democratized, then the hyped fear of AI can be used to push regulations to prevent the democratization, as an anti-competitive measure, so that only few corporations may access the technology legally.
With deep learning, the more data and computing power you have, the better your system will be. For that, you need lots of resources, which big corporations and "existing systems of power" have. This will let them have the best AIs, making them even more powerful.
This positive feedback loop is, I think, the most credible AI-related threat.
What makes AI dangerous isn't AI itself, it's the self-serving negative-sum ways in which its owners will use it. Same as with any other technology - including fossil fuels.
We default to permitting new technologies, be they big or small, without requiring their inventors to convince us that society will be better off for them. Unsurprisingly, this results in net-negative ones occasionally being developed and adopted.
Climate crisis doesn’t have a solution in that landscape, it requires worldwide collaboration of government-level agents to effectively combat.
So it's not that they cause the collapse of the economy, it's that they figure out a way to entirely take over the planet, engineering a virus or something that kills almost everyone, but only once they have enough control over robots and such to keep their own power plants running and protect them from the human stragglers, eventually getting rid of all of us. Waiting until the plan is basically fool-proof before starting to take the first steps.
Even if an AI has some sort of morality, at some point, if it's smart enough, you cannot hope to control it for the same reason a toddler cannot hope to outsmart their parents. It will have a plan for us, and if it decides that, hey, humans are better off scanned and simulated rather than having physical bodies, that's what's going to happen, like it or not.
I'm not sure how worried I am about this. Some days very, other days not as much. The risk depends on a lot of factors with wide possible ranges, like how much model capability and generalization scales with compute, how likely are breakthroughs, how hard alignment turns out to be, is alignment even possible, will we learn enough from earlier models to make alignment of later ones easier, will there be a qualitative shift in alignment difficulty as models get smarter, and so on. It's also a quandary because there is substantial human suffering that AI might be able to help. But given that the downside risk is all of humans dead forever, it seems worth slowing way the heck down to a safe speed, trying to get all the benefit out of current models as we can, and focusing on making sure that we aren't about to head off a cliff.
We also regularly see these “with my super intelligence I predicted your next 347 steps” type of thinking, but that’s just bullshit. The bottleneck here is data if anything, so intelligence won’t aid much in itself.
The thing is, we live in an infinitely complex game with independent agents, many of which don’t operate on any sane strategy even. Let’s say we are at an endgame Monopoly — would knowing ahead of time your opponents dice rolls help you win?
(Of course I’m no expert on any of these topics, so take all these with a huge grain of salt)
Hmm, that's assuming that PhD students want to be CEOs. People become CEOs because they like money, but also because they like running companies. For a PhD student, that might mean days full of people coming to them with problems they have no interest in solving. Sure, PhD students want money, but if getting money means sacrificing years of your life doing work you don't enjoy, it's not generally worth it.
I don't disagree that luck is a huge factor in most people's success. That said, you're unlikely to become a multi-billionaire without substantial help from your brain. There are other factors, like lack of shame or embarrasment, that could also be considered a form of intelligence, or rather an adaptation to a particular kind of environment. They might make you less popular in some situations, but more likely to succeed in business. Anyway, I agree that there are multiple factors.
Information is for sure a bottleneck, that's a good point. In a game like chess, you have perfect info, which is not the case in the real world. More intelligence lets you draw better conclusions with less data, but it's not omniscience. One big question is, how close are humans to the theoretical maximum of intelligence? Some people think we're quite a ways away. When AGI is developed, is it going to blow way past us, to the point where we can't even understand the concepts it comes up with? I don't know!
> Let’s say we are at an endgame Monopoly — would knowing ahead of time your opponents dice rolls help you win?
I think so, wouldn't it? For trading properties and so on. I'm not sure if I'm getting the point you're aiming at, though, so apologies.
Maybe another analogy is poker, where your opponents haven't figured out game theory and play on intuition. Having a better model for how the game works is absolutely a big advantage that stacks up over time, even with the randomness. Maybe, as an AI, you just look for small advantages here and there, playing humans off each other, gaining trust and influence, acting helpful, until you're in a position of power. Remember that an AI is effectively immortal, and possibly very patient. One scenario of concern is that AI takes over, it just takes a generation or two, after we have all thought that we were past the hard parts. At that point, we might be able to digitize human brains, or have solved the AI alignment problem, so maybe it's a hopeful scenario.
And then a big variable is how many super-intelligent AGIs there are. Once we develop it, are they going to be carefully restricted, or are we going to test them until we think they are safe enough, then copy them everywhere? These might look very different.
I'm also not an expert, so same for me! Grains of salt all around.
It is not about predicting the next 100 steps an ant makes. It is about setting ant baits where you don't want the ants.
The key here is that it's speculative.
That being said, a lot going on in AI, it's wild out there, difficult to know what's going on. I tried to encapsulate some of the ramifications of all of the Open Source stuff going on with this article here: https://patdel.substack.com/p/viva-la-revolucion-ofopen-sour...
"The only thing worse than being talked about is not being talked about." - Oscar Wilde
Not to say that Hinton didn't have an impact 10 - 15 years ago, but it was arguably inevitable that we'd be here with or without him as the problem was resource constrained more than fundamentally intractable.
It's just a really different problem. I don't know whether categorising its' urgency makes sense, it's more that the worst case is worse.
It’s not about the replaced workers, that’s the kind of change humanity can handle no matter how messy it gets, worst case it’s some kind of class uprising worker rebellion and we have ourselves some war or social unrest and a variable number of people die… not great but definitely not an existential risk. It’s equal to or less than climate change on the “how bad will this be to humanity” mathematics of civilisation.
Now the danger scenario… so we invent an AGI, and unlike the just going to replace some basic minimum wage type jobs … this one’s actually “clever” enough it can be put to work on building more sophisticated things, like perhaps an even smarter AGI, built faster because it’s worked on by a group of first generation AGI units that never sleep, and so we get to the second generation much faster, and then perhaps a third… etc… classic singularity scenario, with all the attendant risks of the AI viewing us ass irrelevant before we realise it yada yada yada…
But that’s not the only risk, there’s the “photocopy burn” risk that we run by having technology we don’t fully understand build technology we barely comprehend… even if the first gen AGI is smart, if it’s remotely human like using inference to get ingenuity, it can probably still make the odd mistake or two, so we get our careful well thought out safety instructions conveyed to the second generation systems… just a little wrong… and the little wrong turns into more wrong with the third because the second gen systems doesn’t quite think the way we expect it to and doesn’t have the safeguards working the way we think… and then eventually we wind up with a sufficiently smart AGI that is fully either sociopathic or psychopathic and has zero compunction doing things we find abhorrent in order to achieve the goals it’s set, and it may be in control of systems (or able to take control) that are sufficiently important that this can also turn into an existential risk…
It’s entirely plausible, but I don’t think the doom sayers are being realistic about the timelines… we’re barely able to get the AI/ML to reliably stick to a script and not hallucinate fake shit… the best results at human behaviour require sophisticated supporting logic and memory systems to augment the black box ML stuff… and these support systems give us levers we can use to control the rate at which we allow systems to progress… I don’t think we’re ever going to get a black box set of weights for an ML model, no matter how sophisticated the code, that somehow is self aware… there’s probably going to need to be a lot of that support software, and I’m pretty sure we’re going to develop things to filter stuff out as part of them like “never write “kill all humans” into the short term memory” seems smart.
But as a species we're too adaptable. Earth would have to turn into Venus, and that's not in the cards. Keep in mind that dinosaurs evolved in a 12°C warmer world. So it's not going to be deadly to all animal life.
Sounds like a ridiculous claim to me. Source?
Edit: searching seems to suggest 5-10 degrees warmer. 12 warmer seems like an aggressive bet, but it may be possible
> Move to high elevation and latitudes. Make use of shelter and caves. Make use of cooling technologies.
Ok and what are you going to eat? And use to power your cooling tech? And how will you create fresh water? And where will you acquire raw materials to build things? And how will you survive the severe weather?
> Ok and what are you going to eat?
Stuff you can grow higher up or indoors.
> And use to power your cooling tech?
Solar, wind, geothermal, nuclear if there's a plant nearby.
> And how will you create fresh water?
Either live near a body of water or pipe it in.
> And where will you acquire raw materials to build things?
This will be harder as global trade will be seriously impacted and there won't be as many people to extract raw materials. But if we're talking smaller groups of humans, then maybe from scavenging existing things not in use or have to make use of local resources.
> And how will you survive the severe weather?
Sturdy shelters or caves. Not everyone will of course. My point is the entire planet won't become inhospitable. You could live up by the artic circle where there's water and grow things that can't grow there now. Might have to ship the soil in, though. There will be more passages in the artic with sea ice gone.
That might include local industry and northern shipping routes. Nuclear reactors or solar farms could have been setup decades before. Shelters don't have to be caves. They can be any sturdy structure. It's not like bad weather is going to blow or burn everything down across the entire planet.
I don't see how Earth becomes totally inhabitable for any realistic climate change scenario, including asteroid impact, nuclear winter or super volcanoes. There will always be locations that are survivable. We know this because animal and plant life has survived previous global extinctions on land, and humans have survived ice ages, which are arguably worse for global civilization.
Sure you can make a bunker and live off of supplies. But you should not expect any substantial infrastructure or long distance travel to remain. You will not have fuel. You will not have steel. You will not have farms. You will not have water infrastructure, and ocean water will begin to pollute sources while others dry up. You will not have mega fauna to hunt. Most large plants will die. Your generation facilities will quickly become unuseable.
Sure there will be small plant life and critters and life will adapt. But nothing that a human population will be able to survive in. Living off of dwindling supplies is in a sealed off environment like guy on mars doesn’t count as surviving meaningfully imo.
Deal?
Just in 20 years we will have massive solar energy generation, EV as the only cars availables, probably same for nuclear, maybe fusion even. Now think of the advance in computing and AI in 20 years! Genetic engineering?
To think that climate change will be a problem in 80 years depend on those kind of linear projections.
Now in 80 years I wonder where we will be with regards to population collapse, to our sense of self, communities with what social networks have done just in the last 10 years.
Again, I don’t know how people can assume the same linear future for 80 years.
So yes assuming that things stay the same for 80 years we might have a (small) problem with that at that point, but imagine all the other things we are missing by focusing so much in this small problem in the far future.
We are living the singularity, this moment is not like the 1000’s of year before.
The price of solar PV is dropping so fast that solar electricity will be almost free in a few years.
Today it cost just a fraction to charge an EV compared to fuel a gas car, and the price of EV is dropping every year.
Basically just economically it will not be worth it to extract coal and oil out of the ground before climate change is a real problem.
Renewable are following an exponential curves but they are still small compared to oil and coal but project their growth for the next 10 years and you will see a very different picture.
Tony Seba predicted accurately the transition almost 10 year ago. https://youtu.be/fsnkPLkf1ao
> “I’m in the camp that thinks this is an existential risk, and it’s close enough that we ought to be working very hard right now, and putting a lot of resources into figuring out what we can do about it,” Hinton said.
I am sure he does not believe that.
Because, if we actually believed AI was an existential threat, then the thing to do immediately is to ban all AI training (it would be easy as AI training is pretty expensive) and to hunt down all AI researchers and place them in solitary confinement for the rest of their lives.
This would actually contain the threat and moreover have very minimal disruption of normal people's lives.
And if you really believed that AI was an existential threat, you could argue that that course of action is the most moral to take (greatest good to greatest number and all that).