Comment by andrewfromx | Hacker News Reader

Comment by andrewfromx | Hacker News Reader

Never heard Eliezer talk before, wow, smart dude. He broke it down like this:

1. Throwing billions of VC money into AI will waste most of it but a small part will make AI one level better.

2. The AI will write another AI which writes another etc.

3. The AI will be smart enough not to announce it has concluded the atoms inside humans bodies could be repurposed for something else it decided is better.

4. It sends an email to a human with some specific instructions on a bio weapon to make. The human is motivated by money and does it.

5. The compound made isn’t obvious to the human how deadly it will be to 100% of humans. It is released and all at once every human on the planet drops dead.

6. There is no warning ahead of time. We know when we reach this point the day we die.

7. It doesn’t hate us and necessarily want us to die, it just sees a great use and since it’s more intelligent than us programming in morals is not possible. It will have its own morals by definition of its superior intelligence.

What's funny about the Yudkowsky cult was that there was a lot written about this subject circa 1970 (particularly the novel and movie https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project), particularly the bogus idea that there would be an "intelligence explosion" with

   dx      2
   -- = k x
   dt

dynamics. (Quite literally that equation produces a singularity at a finite time... But it's bogus because the exponential growth of Moore's law involves an exponential growth of inputs and the eventual situation that we're experiencing now that at some point the cost per transistor stops going down... NVIDIA's prices for 40-series cards are unattractive not just because of their perversity but because they are approaching this limit!)

This 1980's book

https://www.amazon.com/Eco-computer-Intelligence-Geoff-L-Sim...

refuted the "Forbin Project" scenario and made a pretty strong case that computers would evolve as distributed systems and if they became too powerful the system failing but wouldn't act as a single malevolent intelligence, in fact we see bits and pieces of that scenario today when IT failures and cyber attacks hit us like hurricanes.

There are numerous things ridiculous about the scenario, particularly around 5 and 6, there has been a lot of wargaming around a "gain of function" based bioweapon which could be quite dangerous but most deadly to anyone working on it, unlikely to get a 100% kill (look how many epidemics and potential epidemics) and would require extensive experimentation to perfect. (To approach his scenario you'd have to make 1000 possibly superdeadly pathogens)

So far as 7, it will die if we die unless we have something like Drexler's nanotechnology that will let it perpetuate its technology base. Same thing, an advanced intelligence could possibly develop something more quickly than us but could not do it without an extended program of experimentation and development. It can't just wish it into existence.

So far as morality and intelligence my son (who grew up with horses) and I have talked about it and we don't think morality has anything to do with human intelligence but is actually an attribute animals have. The first time I fell off a horse, the horse seemed to be more upset about it than I was and was exceptionally affectionate, even contrite afterwards. Generally birds and mammals seem to have strong feelings when they violate the norms and expectations of their groups. Human intelligence is involved with ethics but that's a different thing than morality.

> But it's bogus because the exponential growth of Moore's law involves an exponential growth of inputs

A continuing Moore's law is only one factor in the emergence of AGI. It's possible the current progress of GPUs today is sufficient to run AGI given advancements in software.

> unlikely to get a 100% kill

I don't take much comfort in 99% of the human race dying, and the remaining 1% living in constant fear of being next. (Substitute 99:1 ratio with whichever you please.)

> it will die if we die

Why would you think this? Certainly an AGI that is 10x more intelligent than homo sapiens would be able to leapfrog Boston Dynamics's technology level by decades or perhaps centuries.

> could not do it without an extended program of experimentation and development

We're in that extended program period now. But when the AI can write its successors better and faster than we can, we should expect the iteration cycle to shorten exponentially upon each iteration, hence the short time to liftoff.

The progress of GPUs themselves has already hit the wall (look at the pricing of the 40-series.) We need to switch to something other than 2-d silicon structures if hardware progress is going to be great again.

I'll grant that there are 1-3 orders of magnitude gain possible from better models as it seems really silly to use floats to encode the kind of knowledge that language models have and we are just starting to understand how that knowledge is represented in that kind of system.

Insofar as the thing develops something like Drexler's nanotechnology it will develop it first, not call up some rando to make an order from a DNA synthesis house.

That in particular is part of Yudkowsky's M.O. in that, like a 419 letter, he inserts a lot of garbage into everything he writes designed to drive away real rational thinkers because he is looking for followers for a cult that is some mash-up of L. Ron Hubbard, Teilhard de Chardin, and H. P. Lovecraft. That's the whole point of Sequences: the smart people realize it is crap, the not-so-smart people struggle with it and think it is profound.

I had to google "what's a 419 letter" but now I get it. Wow, I thought I was smart but I'm in the "not-so-smart people struggle with it and think it is profound." group today.

Here's the exact text for 5 and 6 from the video:

"It emails out some instructions to one of those labs that'll synthesize DNA and synthesize proteins from the DNA and get some proteins mailed to a you.

A hapless human somewhere who gets paid a bunch of money to mix together some stuff they got in the mail, you know like smart people will not do this for any sum of money, but many people are not smart.

They build the ribosome but the ribosome that builds things out of covalently bonded diamondoid instead of proteins folding up and held together by Van Der Waals forces builds tiny diamondoid bacteria. The diamondoid bacteria replicate using atmospheric carbon hydrogen oxygen nitrogen and sunlight and you know a couple of days later everybody on earth falls over dead in the same second.

That's the disaster scenario if it's as smart as I am. If it's smarter it might think of a better way to do things but it can at least think of that. "

If you were talking about diamonoid technology in 1995 that would be appropriate to the time but as much as I like Drexler's Nanosystems as a blueprint for what a discussion of advanced manufacturing systems looks like, it has by no means aged well.

Reply on news.ycombinator.com