Comment by nopinsight | Hacker News Reader

Comment by nopinsight | Hacker News Reader

Parent Full thread

Our civilization and ability to dominate over other animals largely exist because of human intelligence. An AGI could be a great partner toward unprecedented prosperity, but an AGI not fully aligned with human values may also spur a major catastrophe.

Two promising approaches toward AI Safety I have encountered:

* Make sure the AI is not too certain of itself and will always defer to humans as the final judge. https://www.ted.com/talks/stuart_russell_3_principles_for_cr...

* Get the AI to learn and internalize human values. A possible technique is Inverse Reinforcement Learning: https://thegradient.pub/learning-from-humans-what-is-inverse...

Both are being developed by Stuart Russell among others. See his book here: https://en.wikipedia.org/wiki/Human_Compatible. The latter approach was also discussed on several occasions by Ilya Sutskever.

Many believe AGI will not be realized for at least a few decades hence but no one can really be certain it will not be developed before then.

Since several different approaches together are better than one. Bright minds including those outside the field should propose more ideas for such a significant & unique problem of our time.

I'm not sure what the unique existential threat that AGI poses is - humans are more than capable of spurring major catastrophe all by themselves.

I say we take a chance; if it doesn't work out well we are probably going to stuff it all up without the help of an AGI anyway, and at least something intelligent will remain if it decides to kill all humans.

From the article, would you say we shouldn’t prepare for anything if we suspect that a horde of advanced alien space ships may be landing on earth in a few decades (and possibly sooner)?

My point is in thirty years, due to humankinds inability to work together on a common goal of self-preservation and equality, we'll most likely simply be further down the path of the probable collapse of civilization.

If we knew aliens were coming, we'd probably be best off hoping for their benevolence and mercy and preparing as best we could to deserve it.

I respectfully disagree. You may wish to check out “Enlightenment Now” by Stephen Pinker for a more optimistic perspective on human progress.

Aliens may not share our values. What we perceive as good may not be so for them, nor might they care if we are “good”.

You're going to have to implement a value system into the AGI if it is a fully general AGI. Otherwise, it will just use its exceptional intelligence to defeat its own sensors, take control of its objective function, and essentially wirehead itself.

So an AGI without a value system is (mostly) harmless. If the AGI has a defined value system, it's a lot clearer what that value system is. No "whoops gonna destroy the world to make paperclips" will happen.

(A not-fully general AGI might not be smart enough to defeat its own sensors, yet still be smart enough to do lots of harm. The typical example is grey goo, which isn't smart at all, yet plenty harmful.)

An AGI with a small goal that it can wirehead to max out is not harmless by default. It can always add 9s to its confidence that it isn't being tricked and then secure itself while doing so.

Reply on news.ycombinator.com