Comment by nopinsight | Hacker News Reader

Comment by nopinsight | Hacker News Reader

Parent Full thread

This is why I doubt if efforts like OpenAI will ultimately turn out to be beneficial or catastrophic. I understand their purpose of democratizing the power and benefits of AI technology broadly. However, if their platform can provide a critical component or developmental path to AGI in the future, the unrestrained distribution of the technology is similar to spreading the blueprint of nuclear warheads. It could result in an equivalent of nuclear arms race, but worse--because a small team of capable programmers of any rogue organizations could come up with new improvements.

It is true that we are probably quite far from AGI, but nobody really knows how far. We could be closer than most experts think. Few experts in 2006 would expect deep neural networks to beat a Go champion or be able to describe visual scenes in natural language sentences within a decade. Unexpected things happen, and we are much further behind in terms of preventive technology like Friendly AI. (https://en.wikipedia.org/wiki/Friendly_artificial_intelligen...)

Given the existential risks involved, I would propose more careful handling of the technology. (I agree that opening up most other technologies is usually a positive thing.)

See "Superintelligence" for comprehensive arguments against unfettered development of AGI: https://en.wikipedia.org/wiki/Superintelligence:_Paths,_Dang...

>> the unrestrained distribution of the technology is similar to spreading the blueprint of nuclear warheads.

There's an episode of Pinky and the Brain where Brain invents a magnification ray and gigantises Pinky and himself - so they're now giant rats [1]. They walk about some human metropolis, looming larger than skyscrapers and terrorising everyone. To cut the story short, Pinky happens and eventually the gun shoots everything in the world.

... so Pinky and the Brain are now normal-sized mice again. Or, well, they're giant mice, but everything in the world is also giant, so they're small.

I think of that episode during discussions of the dangers of AGI and how "democratizing" it will eliminate those dangers. The problem is that AGI, like nuclear weapons, is not an equalizing force only- it is also a very destructive force, and so there is the danger that any loon with access to it could blow us all to kingdom come (with nuclear weapons, for sure; with AGI, perhaps).

On the other hand, when nuclear weapons, or AGI, giant rays etc are not democratized we have the situation we're in today: only a few entities have access to them and they basically have free (military) reign... and there is still the danger that some loon might get to a position of power were they can push a button and -blamo.

Which is why people are really worried about this sort of thing. Because once certain discoveries are made, there's no going back and at the same time the road ahead is full of danger.

[1] https://www.youtube.com/watch?v=-fPcqu5aMyw

We already have 7 billion of these H(uman)GI's walking around the world, with guns and dangerous tech. If the AGI will be much smarter, we shouldn't fear it the same we fear a dictator with nukes.

"Smarter" has no nessisary or consequent intersection with "Shares human values" or even "Values human existence"

https://wiki.lesswrong.com/wiki/Paperclip_maximizer

One of the reasons AGI will be so dangerous is the near certainty it will have utterly alien values and intentions.

It seems exceedingly unlikely that we will be able to program in what consitutes "human values". We can't even begin to remotely agree and explain them fully to fellow humans, let alone what is basicly an alien intelligence.

It is slightly ironic that people bring up Assimov's laws of robotics in this context, because pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI.

If something is very much smarter than humans, we cannot possibly hope to constrain it via some kind of faustian bargin agreement. Because in the end, the devil is always smarter than you are.

If an AI kills you from either malice or indifference, or just that you are composed of plenty of paperclip-capable raw materials, you are dead all the same.

> pretty much every story he wrote was about how the laws could be subverted via "loopholing" of sufficently intelligent AI.

The most chilling of these is the "Robot Dreams" short story.

I still think the AGI will have more to fear from people than people from it. We're together on this little boat called earth, 7 billion of us with access to weapons and many of us utterly crazy and willing to do stupid things for stupid reasons (see religion). We don't need AGI to blow ourselves up, plenty capable of that using our wetware neural nets. Imagine being locked in a prison cell with so many dangerous HGIs (Human GI's). I think the first thing for AGI would be to make sure we can't kill ourselves and it at the same time, by teaching us to become less stupid.

Repeat after me, "AI is not Magic."

A super intelligent AI does not change the laws of physics or even get to predict the weather two weeks from now.

Sure, they could learn to be a good hacker but the worlds best hacker can't get into air gaped systems. More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.

I'll admit this is purely hypothetical, but given that we know many people act on stuff they read on the Internet in an uncritical fashion - we cannot dismiss AI because it "doesn't have access to air gapped systems". A human has access to that system - a superintelligent AI could convince that human to "push the buttons".

I don't actually believe this will happen, but I'm not 100% confident it wouldn't. I don't know what a superintelligence would look like!

Then your problem is humans, not AIs. And "pushing a button" is a severe understatement of what an AI would need to have a human do to cause the implied level of damage.

I have read some stories where various airgapped systems were "compromised" in some way (microphone picking up CPU harmonics,etc). See here for example: https://thestack.com/security/2016/06/24/even-speakerless-ai... Obviously those are fringe cases and not practically exploitable outside of a specific environment today.

What I think the danger with a super intelligent AI is being able to correlate massive amounts of data to see patterns we miss or things to try, and never being bored. Look at automated exploits today vs the old "hacking by hand". Brute force attempts not just on known patterns, but being able to create new tests and analysis.

The power of an AGI goes far beyond intelligence. It could generate millions of its own instances (possibly with variations) and hack on thousands of systems at the same time, actively adapting each one as needed. It could perform social engineering on a large number of people simultaneously and communicate in ultra-high bandwidth and protocols that most humans involved cannot follow.

A likely potent form of gaining power may involve assuming the identities of people one knows, via text messaging or email, and use that to gain access to important information. The aggregation and superhuman-level analysis of private information thousands of times faster than humans would bring enormous financial and social power to the AGI. Given the sad state of security in most systems, and the relative ignorance of information security issues among many political and business leaders, an AGI will not have much problem with this sample plan off the top of my head.

And there are many other more clever plans an AGI could come up with that we have not yet considered...

Suppose AI 1 uses that approach but Gmail has AI 2 acting as a spam filter. Suddenly phishing becomes much more difficult.

Further, spinning off other copies requires hardware to run them. If MS has an AI 3 go over windows source code it may be much harder to hack desktops.

Without authorial fiat a world of real world AI's may be much more stable and boring than you might think.

Defense is generally much harder than offense, in most systems. The attacker needs to find only one loophole. The defender needs to protect all of them, including the fallible humans in the loop. This is one reason even large corporations often fail against a small group of hackers.

And in the real world, not everyone exclusively uses services and systems from Google and Microsoft, presumably two of the most competent technology providers. Plenty of people also use ad hoc services set up by internal teams.

VISA, MasterCard, the US Federal Reserve, and IRS etc are a vast targets. However, while people regularly fake CC, Visa's internal systems seem safe. Suggesting defense is actually easier than offence when both sides are competent.

> More importantly, flawless security trumps hacking. If there is nothing to exploit there is no way in, which means a lone AI is much more powerful than lot's of AI's with different goals.

No security is flawless in the face of a sufficiently determined attacker. No system has nothing to exploit.

Airgapped systems have been attacked in the past. Stuxnet is a good example. No magic required.

Just goes to show that flawless hacking trumps security.

There's a subtle argument here philosophically that I think is crucially tied to AI: determinism.

If the world is deterministic, a sufficiently powered and advanced AI with enough data could actually predict the weather, not just for two weeks, but for years. This also enables many other things viewed as magic.

This, of course, is incredibly far off. And I don't see a flaw in your point on flawless security. Though, I doubt we have many, if any, truly flawless security systems in existence. We just don't know their flaws, making them good enough for now. The world could get really interesting once AI's start programming and writing app security to protect itself from malicious AI's.

Even with a totally deterministic model, you can't predict weather very far out because it's chaotic. In other words, you'd need to measure the parameters with arbitrarily high accuracy, which even an advanced AI can't do. To oversimplify, if you need one more bit of accuracy in the measurements for each hour you forecast, it's impossible to get enough accuracy to forecast a week out.

For a detailed discussion of chaos and weather prediction, see: http://www.ecmwf.int/sites/default/files/Chaos%20and%20weath... The "trick" for dealing with chaos is to do a bunch of predictions with perturbed values. If they all end up the same, then the forecast is probably okay. If they vary widely, then you know not to assign much accuracy to the forecast.

I agree that "determinism" has much to do with the questions here, but to immediately pose a question about "the world" being deterministic might be putting the carriage before the horse.

I think the the other, perhaps more prescient point here, is about whether or not that kind of exponentially-learning AGI being discussed is even possible given that the substrate such things would be embedded in (hardware, software) which are [or appear to be?] deterministic. One assumes that that kind of learning implies learning about or aquiring capabilites which have not already been determined or specified by a human. At some level of abstraction, humans _must_ determine that system. It is possible to concieve of a machine that is non-deterministic in that way?

At what point does such a system actually leap beyond what has been specified by us? Does this imply a machine that not only rewrites itsel in softwate, but also remanufactures itself in hardware?

I did some heavy research in various forms of machine learning and AI in grad school 10 years ago, and the more experiments and tools I created, the more I saw this "digital nuclear arms race" and didn't want to be part of it.

We don't know how many other people aren't working on this because of moral/ethical reasons. Of course, 99.9% of the world could be wary of genetic engineering, but that remaining 0.01% is enough to pursue research, get VC investment and drag the rest of us into that uncertain future.

To continue the nuclear analogy, the Manhattan project was probably the most impressive engineering program in the history of the world, but it was driven by survival in a World War. They didn't build and drop the atomic bombs for fun. You'd think that working on a limitless virtual brain should have similarly serious motivations, not just the examples of "how do I drive to X but also shop for Y?" or "is that a monkey?".

I know there are much grander societal goals with A.I. and the world really could become a "better place", but please sell society those goals, not the usual first-world problems. We already have enough people trying to destroy the world without these extra tools.

Perhaps the general availability of AGI is antithetical to the notion of information privacy in the 21st century. And not just for individuals either, but for governments as well. I can imagine that control will only be possible with very deep, widespread monitoring.

It's a little repetitive and slow in places, but I wish more people read that book. It outlines some big issues with AGI. Changed my outlook on AI for sure.

OpenAI doesn't attempt to solve problems that even remotely resemble what you are describing. Hardly anyone claims to be working on AGI because AGI isn't a well defined problem. Its something cranks and the media tout because its excellent clickbait, along the lines of the singularity.

All the efforts of ML solve problems with a concrete input space and a concrete output space. This means they solve very discrete problems and are not generalizable, unless they have similar looking model inputs and the model has the same limited range of "moves" it can make.

I understand the limitations of current ML systems. That is why I mentioned 'a critical component'. Given that humans may use pattern recognition methods broadly similar in principle to deep neural nets, the conjecture is not without merit.

Reply on news.ycombinator.com