Hacker News Reader

Comment by emptysongglass | Hacker News Reader

A lot of HN chatter (and I'm inclined to agree) believe a number of those people involved are pro-safety to defend their moat and promote regulatory capture.

There's a very large contingent of ML researchers who think any idea of AI extinction risk is foolish because we don't have any evidence that intelligence equals compute. I've yet to see a single person give evidence that these two things are equal. What's more, the missing ingredient in any of these AGI extinction scenarios is desire (desire to act, desire to be, desire to love, to kill, etc.) and if you thought there was a paucity of evidence for intelligence = compute just wait till you see the evidence for transformers showing evidence of desire.

There's none. Not a shred of evidence. As ever, it's other human beings that are our greatest extinction risk.

View on HN

Before the first nuclear test, they ran a calculation to make sure that there's no risk of the whole atmosphere chain-reacting and the world ending. The guy who did it said he was like 96% confident in that it won't happen. And they went with it anyway. Took a 4% risk of blowing up Earth.

Is this a reasonable chance to take?

"We don't have any evidence that intelligence equals compute" is worse than "we're 96% confident it doesn't". Ilya Sutskever clearly believes this is a real risk (otherwise he wouldn't have thrown his reputation and wealth to the wind by firing his cofounder yesterday). He is one of the foremost experts. So are two of the "fathers of AI", Hinton and Bengio, who both have no interest in creating a moat and yet signed letters saying "this is an extinction risk, lets treat it like nuclear", one of them quitting his job to be able to say it.

We don't have much evidence for or against, but that's not a great argument against "if it's true, we all die".

I don't think the comparison is fair because chemical reactions and how they work was a part of the model of the theory that nuclear testing might blow up the world. We were dealing with real, testable models of the world all the way up to the nuclear tests.

This is more like, if I smash two fish together in the Large Hadron Collider there's a chance the universe ends. Which no one would tell you with a straight face was remotely possible. But! It's never been done before, so, possible?

We're human beings. One of the things humans do is project. We do it all the time. To the entire animal kingdom, to the gods of our theologies, you get where I'm going with this?

Well now our gods have come to visit us and much like the gods of our fantasies, they're proxies, a mirror to the best and worst of ourselves. Because we still largely operate on fear, we have a camp of end-of-the-worlders who believe these gods are ready to judge us guilty and murderbot us all.

Imagine if we were a species where fear was not the dominant feeling. We'd in all likelihood imagine transformers were harbingers of something else entirely.

Intelligent agents that don't share your worldview can be dangerous. We've seen this a hundred times (e.g. Cortez), I don't think it's controversial or "not part of our model".

"The LLM research project might lead to intelligent agents that don't share our worldview" should not be too controversial either?

You could argue about the odds, but it's very explicitly the stated goal, and they came orders of magnitude than any before them, and most of the founders of the field are doing things like quitting their prestigious jobs or throwing billions of dollars of investor money in their investors' faces, that they have nothing to personally gain from, for the stated reason of trying to prevent this outcome. So we should probably defer to the experts and assume, say, at least 1% that it's a real risk? 10%? It's not fish in the LHC, it's part of one not unreasonable model of what might happen as a result of people spending billions of dollars trying to make it happen.

Edit: I see you've made an important edit. Are we still arguing whether it's possible that OpenAI will succeed in creating intelligent autonomous agents, or only about whether we should fear that eventuality?

> What's more, the missing ingredient in any of these AGI extinction scenarios is desire

Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing?

I mean, we kind of already see the effects of autonomous agents on the world in the form of companies maxizing resource extraction for profit. I don't feel this is going to end well, either (tangent, I also don't feel that life constantly getting better because of technology either).

The missing ingredient for benevolent autonomous agents is a purpose, e.g. the well-being of humans.

> Wasn't that point already refuted by the paperclip maximizer thought experiment long before LLMs became a thing?

I mean, no, because Bostrom hypothesized that an AI given the task of maximizing paperclips would consume all life in its quest to produce paperclips but there's just so many assumptions in this chain of thought, it's not credible (to me).

Here's one portion of Bostrom's quote:

> The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off.

So Bostrom imagines the AI realizing? Ok, hold on there chief, there's about a few hundred priors you haven't addressed.

Better off without humans because humans might switch it off? So now we're talking about human feelings here. Emotions. Fear. There's my other comment down below that addresses that.

As long as we keep projecting our fears and emotions onto transformers we are no further along to understanding the extinction risk of LLMs than soothsayers rooting around in the entrails of dead animals are to understanding the outcome of battles.

I see your point, but the thought experiment still claims that AI risk does not require consciousness, "malicious intent" or a "breakout" of the machine in some sense, no?

All of the PCM scenario - including the part you quoted - are described as a means to the ulterior goal (paperclips). Self-preservance just happens to become a logical part of if. The preconditions for an existential risks are not explained by the thought experiment, it just says that there's no need to assume "consious decisions" or consciousness for it, by providing a counterexample.

So you might disagree with this, but the PCM thought experiment does not assume pain, feelings etc in AI.

Of course this is only a thought experiment, and it assumes literal AGI.

I'm also not arguing that LLMs are AGI.

What I wanted to point out is already mentioned by many others: existential risk (or great harm) doesn't need a suprising "conscious decision", or consciousness.

I mean, machines can cause unpredictable harm already if not operated correctly, all without AI.

I fan absolutely believe this. I'm interested in what BigMentalHealth /s feels about LLMs that can be trained off a knowledgebase like CBT/DBT/IFS and other modalities that are basically defined and representable

You can't insist on a Monopoly where much of the population can't access you (cost/coverage) and caveat that with if they can't get help they have to hurt/cause more destruction/die.

Not crazy about the structure here but you get the meaning.

Honestly inclined to agree. Not to question people's ethics unduly, it sounds exactly like the same logic we use to "protect" drug users from drugs by destroying their lives in advance and inducting them into the criminal justice system as a manner of measured due course. Makes sense, yup yup yup. We know what's best for you.

ChatGPT doesn't do anything you can't already get by searching Brave or Kagi. Just a little more human-centric given its chat formst

I don’t think it’s controversial at all to say that drugs can be dangerous in a way that requires protecting users. You may not agree with carceral drug policies (as you don’t have to agree with every conceivable AI regulation), but would you argue that Purdue Pharma did nothing wrong?

By that logic alcohol and tobacco gotta go wholesale. They are the most destructive and addictive and its not just because they are the only legal options. The mental dichotomy people make in differentiating alcohol/tobacco and DRUGS is asinine. They are the worst and they point out how ridiculous the entire regime is. Its also annoying as fuck there are moron cops setting medical and pharmaceutical manufacturing policy at the national or any level.

Might as well throw caffeine (coffee in particular) into that barrel as well if we are talking about addictive substance many of us casually use daily.

The point is its ridiculous to be wading in all this arbitrary nonsense. Why the heck else would Jefferson have been growing poppies at Monticello if he didn't intend for people to be able to grow and possess opium. Thats some 20th century bullshit after they screwed the pooch with alcohol prohibition TWICE, no?

We don't protect users tho. We make them hit the streets and subject themselves to wacky unregulated clandestine chemistry or doctor shop to get what they want or need exposing them to hefty criminal liabillity.

People using drugs the "wrong way" are bound but not protected, people using the "right way" are protected but not bound.

I know because I've been both, not so much these days

Edit: i, too, enjoy being protected, not bound as I like my drinks Stirred, not shaken

Reply on news.ycombinator.com