> I am trying not to panic, not to sink into a deep despair, but it seems like even if AI doesn't actually kill everyone (which apparently over half the people working on AI think has a good chance of happening!) it's going to screw up things much worse than just the weak AIs that we've used to create the never-ending attention economy. It seems like AI is a fire that has already started to burn and consume us, and we just keep feeding it instead of fighting it. Maybe we create AIs to cure our diseases, then someone uses one to take out the power grid and modern society collapses and hundreds of millions starve. Maybe even if a super-intelligent AI doesn't take over, we rely on them more and more until every important decision is made by AI instead of people, and when bad things start happening, it's too late to undo it. We're going so, so fast, and there's nobody at the helm.
You're just being rational. People who don't understand the risks of AGI are simply not thinking about the problem hard enough. AGI is dangerous in every sense of the word. It has unlimited destructive capability, there's no practical way to stop research, incentives both to create and abuse AGI systems are high, and our ability to control a super-human AGI is extremely limited. There's nothing good about the problem that's facing us, and the typical response you'll receive for raising these concerns, eg, "cars were good so I think super human AGI will be good too" are not arguments but cop-outs. And the more you have these discussions you'll notice those who are optimistic about AGI are simply delusional optimists who have no tangible arguments to make beyond appeals to hopeium.
I've had two recurring nightmares in my life, one is of alien invasion and the other is of a post-AGI world. And when I say nightmares, I mean I wake up in a cold sweat at least once a month from one of these two things and it's been this way since I was a teenager.
I've also been obsessed with AI and the nature of consciousness my whole life. This was the reason I studied NNs at university prior to deep learning revolution – back when no one gave a crap about AI or NNs. I have a blog on the topic of AI, AGI and consciousness going back 15 years and even wrote a book on AI a decade ago. I mean it's the reason I'm on HN today – I got into startups and coding because these were adjacent to my interests in AI.
In some ways I feel like everyone has now just caught up to where I've been on this my whole life. I was reading back on some notes I wrote in my journal from 2012 where I was talking about dying in an alien invasion or in an AGI apocalypse with a stupid amount of certainty. But back then I still thought AGI was some time away.
In hindsight I now think I was way too optimistic. Having some knowledge of AI and knowing how slow progress was in towards AGI I always felt we probably had a decade or two more left. Although where I disagreed with most people working in the field was that in existing architecture and abstractions would probably get us 95% of the way there, and the difficulty of achieving AGI in practise was more in the need for very broad and very large training sets, coupled with huge amounts of compute – but since the launch of GPT-3.5 I've become convinced text is probably enough and that we probably have all the compute needed today to achieve super-human AGI.
The scariest thing about AI when you think about this problem more is that death is actually one of the better outcomes and is actually the one I hope for. Even those who have warned the most about AGI like Eliezer Yudkowsky, Nick Bostrom, Sam Harris, etc, never seem to focus on this point, but I feel a good assessment of risk here would include both the of probability of AGI going wrong and all its potential consequences. It would be like trying to discuss the risk of nuclear weapons in the 60s from the perspective of how likely it is for someone to launch a bomb. The issue isn't that there's some risk that a bomb could be launched, it's that the bomb we're talking about here is nuke. The AGI alignment problem doesn't sound that bad if you don't understand the magnitude of the risks that an unaligned AGI presents.
The primary risk here isn't just that AI might replace humans or cause economic and social chaos, but in the consequences of it being able to solve every difficult problem that is conceivably solvable. The problem of how to read minds... The problem of how to control minds... The problem of how to upload someones consciousness and torture them for 20 million years. The problem of how to push the human body to to the brink of death, without ever allowing it to die. Many of the problems AGI will allow humanity to solve will be those we don't want solved. And to appeal to those with a more limited imagination here, AI will make it trivial for governments (and individuals) to do mass surveillance, build killer robots and create horrific bioweapons like targeted diseases. The space of hard bad problems we never want solved in my estimation is likely to be much larger than the space of hard good problems we'd like solved like how to cure cancer.
AGI basically allows us to create hell and eternal damnation on command – or worse, at its own prerogative. Such things seem silly, perhaps even unthinkable, from our perspective of pleasant 21st century democracies, but this is just our cultural bias speaking. Humans have sought to maximise personal power and have historically taken enjoyment in revenge and the suffering of those they believe "deserve it". The idea AI wouldn't be used for horrific things in my opinion is just an extremely naive and ignorant opinion, and even if by some miracle this doesn't happen in the West you should still consider ways in which other countries will likely use AGI to cause unthinkable levels of human suffering.
But remember, AI has the ability to disrupt existing social orders (and likely will) so it would be a mistake to take comfort in the protection of democracy here. AGI has the ability to make Sam Altman king of the world and ensure that everyone who doesn't follow his every word for the next 1,000,000 years of their life will be tortured for 20,000,000 years. AI makes these futures not just possible, but in my opinion somewhat likely given humans tend to seek power.
So I guess to answer your question, I actually hope to die and take some comfort in that still being an option. My AGI nightmares would stop immediately if I knew I and everyone I loved was just going to die. My fears all come from the fact that I don't want to see or know what kind of hells are possible in a post AGI world.
The problem I have now is that I've long argued that the minute AGI is created the safest bet would be to immediately commit suicide since the risk of death from AGI would be is high anyway and the tail risk of extreme suffering is high enough that death is both the safest and reasonable option.
Anyway hope I'm just a lunatic.