Hacker News Reader

Comment by edw519 | Hacker News Reader

My first reaction was that if you're going to live forever, why rush to graduate, there's plenty of time. Then when I read the article, I saw that they are looking at all kinds of other opportunities. Good stuff.

View on HN

This isn't a person attack. It's just a general observation.

People seem to vastly oversimplify Kurzweil's (and others) stance on the singularity. The truth is that we don't know what will happen. The best one can do is prepare for possible senarios.

Not everyone expects that the singularity will be survivable. Extinction still seems the most probable outcome.

this is why I don't understand the "nerd utopia" line. it seems that the vast vast majority of possible non-human intelligences would have goals directly harmful to human goals (even modest ones). It seems like the best we can hope for is a really nice zoo.

I would say that the best we can hope for is that intelligence is so hard to understand that we get uploading first, and therefore the vast majority of non-human intelligences are previously-human intelligences -- that is, us.

The second-best possibility is that intelligence is possible without consciousness ("really powerful optimization processes") and no one creates people AI before we are adept at using RPOPs to further our human goals.

Both of these seem like wishful thinking, unfortunately. I don't have any idea how to push the future toward one of these outcomes.

If you said that about a topic you're familiar with, it sounds silly: it seems that the vast vast majority of possible computer programs would have goals directly harmful to human goals (even modest ones). The vast majority of possible medical treatments, proteins, DNAs, etc. would be directly harmful to human goals, and yet we keep on trying, cognizant and wary of these ramifications.

Granted, there are "irresponsible" researchers who don't understand that an AI might not have "gratitude" that prevents it from rm -rfing humanity like a bad shell script, but others work on 'Friendly AI' to mitigate or eliminate this concern.

http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...

A recursively self-improving intelligence seems far more likely to be an existential threat than virtually any other human creation. Even gray goo can be fought fairly easily with heat (yes, "fight it with fire" will be the slogan of the First Goo War... ;)), but if a hard takeoff is possible, the first AI will have nothing to oppose whatever its goals happen to be. The touted solution to this is Friendliness, as you note, but Friendliness appears to be a problem in the same class of difficulty (or more so) than AI itself, even if workable at all, which is uncertain. In my opinion, people who choose to work on Friendliness as a prerequisite to AI are simply guaranteeing that AI will be developed by someone else first.

Which is why this is important:

http://en.wikipedia.org/wiki/Friendly_artificial_intelligenc...

the things you mentioned would be harmless, not harmful. at worst they would waste some resources. they certainly wouldn't requisition resources by force (unless you want to talk about ideology as a "technology" :)

Er, prions, viruses, bacteria, runaway breakout of monoculture -- these things use up resources, and have the potential to cause great harm. It's just that few of them are likely to rise to the level of existential threat, while in the field of recursively self-improving AI, the likelihood is strongly the other way.

http://en.wikipedia.org/wiki/Morris_worm

http://online.sfsu.edu/~rone/GEessays/killervirus.html

I think the vast majority of possible non-human intelligences would simply leave earth

It wouldn't just come into being by chance (if we make it). We'd write its source code, which is why FAI is important.

Reply on news.ycombinator.com