Hacker News Reader

Comment by wongarsu | Hacker News Reader

A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are almost certain to occur, but also for the most part have relatively minor consequences (mostly making things that are already happening worse). "Advanced AI will make humans extinct" is much less likely but has catastrophic consequences. Focusing on the catastrophic risks isn't unreasonable, especially since society at large seem to already handle the more frequently occurring risks (the EU's AI Act addresses many of them).

And of course research into one of them benefits the other, so the categories aren't mutually exclusive.

View on HN

This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow.

We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important.

Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions.

Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.

Today's real issues are not that important compared to human extinction.

I’m not sure if extinction is a problem either. No one’s left to care about the issues, then.

How long would the currently 17 people in space survive if every human on planet earth was eradicated?

Dunno. I imagine they’d follow soon, though.

It seems to me that the most viable routes to human extinction are through superscaled versions of contemporary disasters: war, disease, and famine.

Although war, disease and famine may kill many people, they are extraordinarily unlikely to cause extinction per se. The human species has so many individuals that only very special risks are capable of reducing 8 billion to zero.

For most mammalian species if they have a stable 100k individuals then they are doing pretty well. So we have 80000x more humans than what would be considered a healthy population in other species.

The risks of human extinction tend to be more long-tail events that have a planet wide effect. For example, a large impact, supervolcano eruption, or engineered bio weapon. AI certainly deserves its place on that list.

I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues.

As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing.

Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.

This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.

At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong

1: https://en.wikipedia.org/wiki/Pascal%27s_mugging

Is AI Safety a Pascal's Mugging?

[Robert Miles AI Safety]

https://www.youtube.com/watch?v=JRuNA2eK7w0

This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...

Existential risk is one of those problems that nearly impossible to measure in most cases.

In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.

But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.

Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.

Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite.

If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.

If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care.

Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).

So the following requirements:

1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century

2) AI acquires power/means to effect human extinction

3) AI decides against coexistence with humans

Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...

Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.

Reply on news.ycombinator.com