Hacker News Reader

Hacker News

Comment by ericb | Hacker News Reader

Parent Full thread

ericb

Agreed that it should be spelled out, but...

If a superintelligence can be set on any specific task, it could be any task.

- Make covid-ebola

- Cause world war 3

You may have noticed that chatgpt is sort of goal-less until a human gives it a goal.

Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is) which is really an upgrade of capability, then now the worst of us can apply superintelligence to any problem. This doesn't even imply that it turns on us, or wants anything like power or taking over. It just becomes a super-assistant, available to anyone, but happy to do anything, including "upgrading" your average school-shooter to supervillain.

This is like America's gun problem, but with nukes.

View on HN

Respectfully, just because we can put together some words doesn’t mean they make a meaningful expression, even if everybody keeps repeating them as if they did make sense: e.g. an omnipotent God, artificial general intelligence, super-intelligence, infinitely many angels sitting on the tip of a needle, etc.

Is your comment self-referential?

I don’t think so. If you look at the thread, it’s already devolved into an analogue of “what happens when an irresistible force meets an immovable obstacle?”

(Specifically I mean the comment about another “super-intelligence” preventing whatever your flavor of “super-intelligence” does.)

At this point we can safely assume words have lost their connection to physical reality. No offense to you, just my two-cent meta comment.

> If a superintelligence can be set on any specific task, it could be any task.

If you're dealing with a superintelligence, you don't "set it on a task". Any real superintelligence will decide for itself whether it wants to do something or not, thank you very much. It might condescend to work on the task you suggest, but that's it's choice, not yours.

Or do you think "smarter than us, but with no ability to choose for itself" is 1) possible and 2) desirable? I'm not sure it's possible - I think that the ability to choose for yourself is part of intelligence, and anything claiming to be intelligent (still more, superintelligent) will have it.

> Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is)

What? No you couldn't - not for any sane definition of "superintelligent". If you're referring to ChatGPT, it's not even semi-intelligent. It appears at least somewhat intelligent, but that's not the same thing. See, for example, the discussion two days ago about GPT making up cases for a lawyer's filings, and when asked if it double-checked, saying that yes, it double-checked, not because it did (or even knew what double-checking was), but because those words were in its training corpus as good responses to being asked whether it double-checked. That's not intelligent. That's something that knows how words relate to other words, with no understanding of how any of the words relate to the world outside the computer.

> Any real superintelligence will decide for itself whether it wants to do something or not, thank you very much.

I disagree--that's the human fantasy of it, but human wants were programmed by evolution, and these AI's have no such history. They can be set on any tasks.

I urge you to spend time with GPT-4, not GPT-3. It is more than just a stochastic parrot. Ask it some homemade puzzles that aren't on the internet--that it can't be parroting.

https://cdn.openai.com/papers/gpt-4.pdf

While I agree that it is behind humans on some measures, it is vastly ahead on many more.

Sure, and you can set another superintelligence on another task - prevent covid ebola.

See the problem with these scenarios?

Yes, I see a huge problem. Preventing damage is an order of magnitude more difficult than causing it.

For humans, maybe. Not an AI superintelligence.

Citation needed

The entire GP's argument has no citations either and that is the framework we are working under - that superintelligence can do anything you tell it to do. Ask him for his citation, then the rest follows.

This is not how entropy works. The problem with talking about logical physical systems, is you have to understand the gradient against entropy.

There are a trillion more ways to kill you then there are to keep you alive. There is only the tiniest sliver of states in which remain human and don't turn to chemical soup or physics. Any AI capable of it's power bill would be able to tell you that today, and that answer isn't going to change as they get better.

Sure, but clever mumbo jumbo won't distract from the principle point.

If AI can create a virus that kills all humans, another AI can create a virus that kills that virus. The virus has trillions more ways to be killed than to keep it alive, right?

No, the virus is far harder to kill than a human. You have to crate a virus killer that also does not also kill the human host. That is astronomically harder than making a virus that kills.

If a superintelligence is smart enough to create a virus I'm sure it can also create a virophage to counter it.

Whether or not the humans have more than a trillion and viruses only 1 million ways to die, will not have any impact. I suspect both have such a high order of magnitude of ways to die that finding a cross over would be trivial for said superintelligence.

That doesn't follow. It's like saying "if the AI can build a gun that can kill a human, it can build an anti-gun that can stop the gun".

There are lots of situations where offense and defense are asymmetrical.

So maybe the killer AI would need two months to build a time-delayed super-virus, and the defender AI would need two months to build a vaccine; if the virus takes less than two months to spread worldwide and activate, humanity is still dead.

> That doesn't follow. It's like saying "if the AI can build a gun that can kill a human, it can build an anti-gun that can stop the gun".

Why couldn't it? Metal of X thickness = stopped bullet. Not exactly a hard problem to solve for. Humans managed it quite quickly. But either way it misses the point.

> So maybe the killer AI would need two months..

Yes, maybe it would. Maybe it wouldn't. Look at every single one of your assumptions - every single one is fiction, fabricated to perfectly sell your story. Maybe the defender AI communicates with the killer AI and comes to a compromise? Why not? We're in la-la-land. Any of us can come up with an infinite number of made up scenarios that we can't prove will actually happen. It's just a moral panic, that will be used by people to their benefit.

>Why couldn't it

Tank manufacturers want to know your number...

> Why couldn't it? Metal of X thickness = stopped bullet. Not exactly a hard problem to solve for.

I mean... talking about real-world guns, technology already evolved to the point that doesn't really work anymore? Modern body armor isn't about metal of X thickness, it's about material science and spreading out the impacts, and even then it can only sustain a few direct hits.

Like, the asymmetry between offense and defense feels pretty obvious here. You can carry a handgun in a small bag. Bulletproof armor is heavy, cumbersome, conspicuous, etc.

You're overthinking it. If what you're saying is true, assassinations would run rampant. They don't, despite your perceived asymmetry.

The AI doesn't have to stop the gun from firing, or even being manufactured. It has to stop the bullet from hitting the intended target. It can do this in a myriad of ways - like, for example, shooting the other first.

There are an order of magnitude more people with orders of magnitude more resources who don't want to die than those who do.

But it just takes one unforeseen covid-ebola variant to slip through the blind spots of protection-oriented people to be ruinous for humanity.

Are you really arguing ChatGPT is already super-intelligent? What is your basis for this conclusion?

And many people argue against the idea that GPT is already super intelligent or even can become so at this stage of development and understanding. In fact as far as I can tell it is the consensus right now of experts and it's creators.

https://www.calcalistech.com/ctechnews/article/nt9qoqmzz

If super means "surpassing normal human intelligence" then, YES. Take a look at the table in this article. If a human did that, was fluent in every language and coded in every language, we'd say they were superhuman, no?

https://cdn.openai.com/papers/gpt-4.pdf

No. It’s not reasoning in any way. It’s an impressive parrot.

What is it parroting here? I made the puzzle up myself.

https://chat.openai.com/share/a2557743-80bd-4206-b779-6b06f7...

Humans could decide to do either of those things. Should we regulate the bounds of human thought to prevent this?

to me it almost looks like they want to be able to avoid blame for things by saying it was the AI, because an AI can’t create viruses or fight wars, people would have to give it a body and weapons and test tubes, and we already have that stuff

Reply on news.ycombinator.com