Hacker News Reader

Comment by ftxbro | Hacker News Reader

If anyone can't get the article, this one is by Eliezer Yudkowsky. He has pretty much devoted his life to thinking about how to not let AI get powerful and destroy all humans, and now there is this wild race among the largest corporations and possibly governments to be the first to get the most powerful superhuman AI system. Obviously he is disturbed by such a development.

View on HN

This is the one moment he has been waiting his entire life for.

He literally waited his entire life for this moment, obviously he's gonna milk it to every drop.

Self-aggrandizement isn't anything foreign to Eliezer, but he's genuinely terrified. Milking it for every drop doesn't really capture what he's doing.

Yeah, I wonder sometimes how many people gloss over what it would mean for an “ASI” to exist. We would - if successful - effectively be creating a supernatural being (at least, there wouldn’t be much of a distinction between it and one).

Sure, there’s hype, and FUD, and fatalism, etc. But, if you believe the creation of such a thing is within your lifetime then it would be difficult to find many higher priority issues to prepare for/help solve/vote on/etc.

In reality, we all likely still downplay the risk by assessing the limit on the downside as a relatively quick extinction of life on earth. There are many things one can imagine might be a lot worse than death.

GPT-4 is superhuman in some ways.

In the same sense as a Sinclair ZX80 was.

GPT-4 can tell you where you’re wrong, Sinclair ZX80 cannot.

GPT-4 is an unreasonably effective autocomplete. It's not telling you "where you're wrong". CMD-F in Notes.app tells mhe where I'm wrong in the same sense, as long as I've written the right answer down before.

The degree of blithe skepticism you express is difficult to understand.

A Sinclair? Please explain what you mean by this comparison.

Being forced to read terrible rationalist Harry Potter fanfics over and over again

Agreed this is not discussed often - in a doom scenario, a quick death for everyone might be the best case

- AGI is here we're all gonna die

- ΞＬＩΞＺΞＲＣＨＩＬＬＯＵＴ

- They will destroy us all

- Ｉ＇ＭＮＯＴＧＯＩＮＧＴＯＤＯＴＨΛＴ

- This may be my last message before they get me

- Ｉ＇ＭＪＵＳＴＮＯＴＴＨΛＴＩＮＴＯＹＯＵ

- AAAAAAH

"he's gonna milk it to every drop" is such an odd take.

eliezer has long been concerned about AI and the risks it poses to humanity. and for just as long people have called him crazy and made hand-waving arguments for why we shouldn't be concerned.

now we're in the midst of an AI arms race and we don't have any good idea how this tech works. it progresses at a truly astonishing rate, where it's become sport to find instances of people saying "AI will never be capable of X" and showing them the latest AI doing X with ease.

i think his concern is real and justified. you might disagree, but i don't understand why think he's milking recent developments.

> It's become sport to find instances of people saying "AI will never be capable of X" and showing them the latest AI doing X with ease.

Interesting, I've not seen that many educated in technology make that claim that it will never, just that people are surprised that the folks leading this, Microsoft and Google, have a track record of turning their consumer facing products to advertised junk.

I've seen plenty of people making nearly those claims. Barely qualifying things with phrases like "not in my life."

Especially in the arts. Granted, that's dropped dramatically over the last decade since ML started taking off.

> we don't have any good idea how this tech works.

Do you mean the specifics of GPT-4, or Transformers in general ?

Presumably he means all types of machine learning in general.

he's likely talking bout the internals. sure we know how to train them but nobody knows what the models learn exactly. how those billions of parameters shape the output on inference.

a few months ago, just this year some researchers discovered what might be the neuron that largely decides when to use an in gpt-2. yes 2. that's what he means.

https://clementneo.com/posts/2023/02/11/we-found-an-neuron

The thing is, there's no evidence that the current path will lead to superintelligence. It could be the case that despite feeding a trillion dollars in compute / data to a model, that we still need to hold it's hand to do anything useful.

He jumped the gun and he's really tarnished his reputation. How can anyone take him seriously after the insane rhetoric and hyperbole of this article?

That's besides the point. The argument still holds. We don't want to find ourselves with our foot on an exploding mine before we stop walking across the minefield. At that point it is too late.

> The thing is, there's no evidence that the current path will lead to superintelligence.

I don't understand how any person paying attention can think this. Just watching the jumps from GPT-2 in 2019 to GPT-4 today makes it clear as day we are rapidly drastically improving capabilities and there's no evidence we will hit a wall any time soon

How can you see the clear jumps in intelligence from GPT-2 to 3 to 4 and not only not believe that this leads to superintelligence, but see no evidence that this leads to superintelligence?

Thinking about or thinking about thinking about, does he have any credentials at all other than basically being early

What credentials beyond "thinking about this problem a lot" would satisfy you? He works at his own research institute dedicated to this topic, wrote some books on it and other (self proclaimed) alignment experts take him seriously.

I think he's a pseud with no real connection to AI research.

I'm not saying he's stupid or anything like that I just don't see any useful scientific output from him.

I think he worked with Kurzweil and Ben Goertzel at some point.

I mean, his whole thing is "progress on AI research leads to the apocalypse." He's done some work on decision theory, but they're not connected to DL, presumably because he thought "well if DL works out in the short term we're all dead anyways." Which seems to be what's happening.

and possibly terrorist organizations, cartels, hacker gangs or just a lone wacko.

Reply on news.ycombinator.com