Hacker News Reader

Comment by bootsmann | Hacker News Reader

> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent.

I have yet to see anyone who argues this point show sufficient knowledge of the current state of AI research to credibly make it. If you live in the fantasy world where AI is just an arbitrary construct that can do whatever task by itself it is an easy point to make but anyone who is down and getting their hands dirty in the pile of linear algebra that makes up current AI knows that there is not even a credible pathway to creating an AGI in current research (i.e. you cannot optimize for a task you don't know at train time and if you are optimizing for a specific task you are not creating a general AI).

View on HN

> If you live in the fantasy world where AI is just an arbitrary construct that can do whatever task by itself it is an easy point to make but anyone who is down and getting their hands dirty in the pile of linear algebra that makes up current AI knows that there is not even a credible pathway to creating an AGI in current research (i.e. you cannot optimize for a task you don't know at train time and if you are optimizing for a specific task you are not creating a general AI).

Humans are optimized for the specific task of reproducing, and yet we evolved general intelligence. "General intelligence" doesn't have to be a goal for it to emerge.

How about a simple example - train a model on outputting code which:

- scans open network ports through google dns

- copies itself and the model to that location

- whenever it detects a sigkill or file delete, scan again and save itself somewhere else

Now you've got an autonomous virus that will never die. Add any other power-seeking and self-replicating strategies using AI modules - the task could be "generate code to run as many duplicate processes as possible" or "generate code to find other floating AI modules like me and share model weights with them". Codex might not be there yet but we're getting close.

Can I tell you something secret? The "AGI" of the form you are describing is already here! Not only that, it's been here since 1988! Crazy right? Just check out this link if you don't believe me: https://en.wikipedia.org/wiki/Morris_worm

(Snark aside, I've realized this type of interactions have always resulted in a very low perception of EA for me. I don't come out of these conversations thinking "what a clueless thing to say by this person who also happened to support EA", I think "wow, EA is full with looney bins who don't know what they are talking about." If you are really into giving what you can and aren't sure why EA gets a bad rep, look no further.)

I've heard of some of the famous viruses which are obviously more complicated than what I mentioned. But that's the difference - they are hardcoded to perform these particular steps. I'm not saying AGI is the code that does this. AGI is one level higher - a more meta objective - given the objective of "accumulating power", it will learn lower level objectives like "replicating and spreading", which leads to the final output like this code.

Basically, AGI could unleash a thousand "morris worms" - and that would only be for the objective of replicating and spreading. There could be other objectives as well.

Reply on news.ycombinator.com