Comment by karmacondon | Hacker News Reader

Comment by karmacondon | Hacker News Reader

This is a key takeaway: "...we are going to ask YC companies to make whatever data they are comfortable making available to OpenAI. And Elon is also going to figure out what data Tesla and Space X can share."

Money is great, openness is great, big name researchers are also a huge plus. But data data data, that could turn out to be very valuable. I don't know if Sam meant that YC companies would be encouraged to contribute data openly, as in making potentially valuable business assets available to the public, or that the data would be available to the OpenAI Fellows (or whatever they're called). Either way, it could be a huge gain for research and development.

I know that I don't get a wish list here, but if I did it would be nice to see OpenAI encourage the following from its researchers:

1) All publications should include code and data whenever possible. Things like gitxiv are helping, but this is far from being an AI community standard

2) Encourage people to try to surpass benchmarks established by their published research, when possible. Many modern ML papers play with results and parameters until they can show that their new method out performs every other method. It would be great to see an institution say "Here's the best our method can do on dataset X, can you beat it and how?"

3) Sponsor competitions frequently. The Netflix Prize was a huge learning experience for a lot of people, and continues to be a valuable educational resource. We need more of that

4) Try to encourage a diversity of backgrounds. IF they choose to sponsor competitions, it would be cool if they let winners or those who performed well join OpenAI as researchers at least for awhile, even if they don't have PhDs in computer science

The "evil" AI and safety stuff is just science fiction, but whatever. Hopefully they will be able to use their resources and position to move the state of AI forward

'The "evil" AI and safety stuff is just science fiction, but whatever.'

umm... you can offer proof that we have nothing to worry about?

Does the proof go like: Just as all people are inherently good, therefore all AIs will be inherently good?

Or is it more like: since we can now safely contain all evil people, therefore we will be able to safely contain evil AIs?

Sounds to me like there is some risk, no?

I thought of responding to that bit too, changed my mind, but your response made me reconsider again. The response is simple, just a quote from Yudkowsky: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." When all one knows about potential future AIs comes from sci-fi, and the only sci-fi one reads or watches is weak sci-fi with very human good/evil AIs, one's frame of reference in the discussion about whether concerns are warranted or not is way too narrow to be worth any further consideration.

The burden of proof is on the people warning us of the impending AI apocalypse. The fact is we are nowhere close to AI. We don't understand how the brain works. A review of the ML literature will also show we barely understand how neural nets work.

I believe in the precautionary principle which says the exact opposite.

"The precautionary principle ... states that if an action or policy has a suspected risk of causing harm to the public or to the environment, in the absence of scientific consensus that the action or policy is not harmful, the burden of proof that it is not harmful falls on those taking an action." [0]

[0] https://en.wikipedia.org/wiki/Precautionary_principle

If we adopted really strict adherence to that rule as the bar to research, there would be no scientific progress at all. I'm not convinced that would be a desirable thing.

well, the clever solution here is not to demand a stop to all AI research, but rather to speed it up to reduce the chance that a single bad actor will get too far ahead... i.e., to get "post-singularity" ASAP, and safely.

Definitely bold... might be just crazy enough to work! Would love to see the arguments laid out in a white paper.

Reminds me of the question of how far ahead in cryptology is the NSA compared to the open research community.

You haven't shown the precautionary principle is the right principle to follow, you've only invoked it.

The fact that we don't understand how neural nets work despite the excellent results makes an AI apocalypse more likely, not less. This means that if we ever create strong AI, we will likely not understand it initially.

Note: I'm personally not too worried about the AI apocalypse, but I think "we don't even understand neural nets" should cause more concern, not less.

True AI will require several breakthroughs. The fact that we don't understand neural nets means that much of the progress thus far has been hand-wavy and hacky iterative improvement (engineering, vs. theory). This means progress is very likely to plateau. When everything is hacky and you don't understand what's going on, those breakthroughs are not going to happen.

As I've said many times on HN over the years, there is currently no clear path to science fiction like "AI". To return your question, hopefully without being rude, is there any proof that AI capable of having a moral disposition will ever exist?

Andrew Ng (I believe) compared worrying about evil AI to worrying about overpopulation on Mars. Which is to say, the problem is so far off that it's rather silly to be considering it now. I would take it a step further and say that worrying about the implications of AGI is like thinking about Earth being overpopulated by space aliens. First we have to establish that such a thing is even possible, for which there is currently no concrete proof. Then we should start to think about how to deal with it.

Considering how hypothetical technology will impact mankind is literally the definition of science fiction. It makes for interesting reading, but it's far from a call to action.

Improvements in AI aren't linear, though. Artificial Superintelligence, after reaching AGI, might happen in the span of minutes or days. I imagine the idea here is to guide progress so that on the day that AGI is possible, we've already thoroughly considered what happens after that point.

Secondly - it's not necessarily about 'evil' AI. It's about AI indifferent to human life. Have a look at this article, it provides a better intuition for how slippery AI could be: https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-...

> Improvements in AI aren't linear, though

This is a point everyone makes, but it hasn't been proven anywhere. Progress in AI as a field has always been a cycle of hype and cool-down.

Edit (reply to below). Talk about self-bootstrapping AIs, etc. is just speculation.

Sure, though you can't extrapolate future technological improvements from past performance (that's what makes investing in tech difficult).

Just as one discovery enables many, human-level AI that can do its own AI research could superlinearly bootstrap its intelligence. AI safety addresses the risk of bootstrapped superintelligence indifferent to humans.

>Just as one discovery enables many, human-level AI that can do its own AI research could superlinearly bootstrap its intelligence.

Of course, that assumes the return-on-investment curve for "bootstrapping its own intelligence" is linear or superlinear. If it's logarithmic or something other than "intelligence" (which is a word loaded with magical thinking if there ever was one!) is the limiting factor on reasoning, no go.

Hype then cool-down is a sine wave... See, not linear at all!

I don't see why a program needs to be a self-improving Charles Stross monster to have an impact on the world, for good or ill.

Or maybe the first few AGIs will want to spend their days watching youtube vids rather than diving into AI research. The only intelligences we know of that are capable of working on AGI are humans. We're assuming that not only will we be able to replicate human-like intelligence (seems likely, but might be much further away than many think), but that we'll be able to isolate the "industrious" side of human intelligence (not sure if we'll even be able to agree on what this is), enhance it in some way (how?), and that this enhancement will be productive.

But even if we can do all that any time soon (which is a pretty huge if), we don't even know what the effect will be. It's possible that if we remove all of the "I don't want to study math, I want to play games" or "I'm feeling depressed now because I think Tim's mad at me" parts of the human intelligence, we'll end up removing the human ingenuity important to AGI research. It might be that the resulting AGI is much more horrible at researching AI than a random person you pull off the street.

is there any proof that AI capable of having a moral disposition will ever exist?

Why does an AI need to be capable of moral reasoning to perform actions we'd consider evil?

The concern is that computers will continue to do what they're programmed to do, not what we want them to do. We will continue to be as bad at getting those two things to line up as we've always been, but that will become dangerous when the computer is smarter than its programmers and capable of creatively tackling the task of doing something other than what we wanted it to do. Any AI programmed to maximize a quantity is particularly dangerous, because that quantity does not contain a score for accurately following human morality (how would you ever program such a score).

If you're willing to believe that an AI will some day be smarter than an AI researcher (and assuming that's not possible applies a strange special-ness to humans), then an AI will be capable of writing AIs smarter than itself, and so forth up to whatever the limits of these things are. Even if that's not its programmed goal, you thought making something smarter than you would help with your actual goal, and it's smarter than you so it has to realize this too. And that's the bigger danger - at some unknown level of intelligence, AIs suddenly become vastly more intelligent than expected, but still proceed to do something other than what we wanted.

The main question is not about whether the AI would or could have morality. The more important question (and I don't think we disagree here) is whether there could be a superhuman AI in the near future - some decades for example - that might "outsmart" and conquer or exterminate people.

This is a matter of conjecture at this point: Andrew Ng predicts no; Elon Musk predicts yes.

I agree with you that, if you can be sure that superhuman AI is very unlikely or far off, then we have plenty of other things to worry about instead.

My opinion is, human-level intelligence evolved once already, with no designer to guide it (though that's a point of debate too... :-) ). By analogy: it took birds 3.5B years to fly, but the Wright brothers engineered another way. Seems likely in my opinion that we will engineer an alternate path to intelligence.

The question is when. Within a century? I think very likely. In a few decades? I think it's possible & worth trying to prevent the worst outcomes. I.e., it's "science probable" or at least "science possible", rather than clearly "science fiction" (my opinion).

We assume that we'll be able to replicate human level intelligence because we'll eventually be able to replicate the physical characteristics of the brain (though neuroscientists seem to think we're not going to be able to do this for a very long time). Superhuman intelligence, though - that's making the assumption that there exists a much more efficient structure for intelligence and that we (or human intelligence level AIs) will be able to figure it out.

So returning to your Wright brothers example, it's more like saying: "It took birds 3.5B years to fly, but the Wright brothers engineered another way. It seems likely that we'll soon be able to manufacture even more efficient wings small enough to wear on our clothes that will enable us to glide for hundreds of feet with only a running start."

>I would take it a step further and say that worrying about the implications of AGI is like thinking about Earth being overpopulated by space aliens. First we have to establish that such a thing is even possible, for which there is currently no concrete proof.

Given that I could probably sketch out a half-assed design for one in nine months if you gave me a full-time salary - or rather, I could consult with a bunch of experts waaaaaay less amateurish than me and come up with a list of remaining open problems - what makes you say that physical computers cannot, in principle, no matter how slowly or energy-hungrily, do what brains do?

I'm not saying, "waaaaah, it's all going down next year!", but claiming it's impossible in principle when whole scientific fields are constantly making incremental progress towards understanding how to do it is... counter-empirical?

Jeez I had no idea you were such an AI genius. If only someone would fund you!

Ahhh... The power of not knowing what you don't know.

I mean, why can't I live forever? Let's just list the problems and solve them in the next year!

Yes, let’s.

http://sens.org

>Ahhh... The power of not knowing what you don't know.

Ok: what don't I know, that is interesting and relevant to this problem? Tell me.

>I mean, why can't I live forever?

Mostly because your cells weren't designed to heal oxidation damage, so eventually the damage accumulates until it interferes with homeostasis. There are a bunch of other reasons and mechanisms, but overall, it comes down to the fact that the micro-level factors in aging only take effect well after reproductive age, so evolution didn't give a fuck about fixing them.

>Let's just list the problems and solve them in the next year!

I said I'd have a plan with lists of open problems in nine months. I expect that even at the most wildly optimistic, it would take a period of years after that to actually solve the open problems and a further period of years to build and implement the software. And that's if you actually gave me time to get expert, and resources to hire the experts who know more than me, without which none of it is getting done.

As it is, I expect machine-learning systems to grow towards worthiness of the name "artificial intelligence" within the next 10-15 years (by analogy, the paper yesterday in Science is just the latest in a research program going back at least to 2003 or 2005). There's no point rushing it, either. Just because we can detail much of the broad shape of the right research-program ahead-of-time, especially as successful research programs have been conducted on which to build, doesn't mean it's time to run around like a chicken with its head cut-off.

It's already here however. Components of drones killing people are AI/ML backed. The data on where to bomb can be "inferred" from various data...

Now for a less "killer" use case, you might get denied access to your credit card because of what Facebook "thinks" based on your feed and your friends feeds (this is a real product.)

AI doesn't have to be full blown human-like and generalizable to have real world implications.

This is what my piece called personas is about.. Most people don't understand the implications of what's already happening and how constrains of programming/ML lead to non-human like decisions with human-like consequences. http://personas.media.mit.edu

As the history of science & technology shows, by the time there is any proof of the concept of a technology lethal to the human race, it is already too late.

I would suggest you read the history of the Manhattan project if you want to continue in your belief system regarding "impossible" deadly technology.

>I would suggest you read the history of the Manhattan project if you want to continue in your belief system regarding "impossible" deadly technology.

To quote Carl Sagan:

>They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown.

How are you estimating how far away AI is so accurately that you can disregard it entirely? The best we can do to predict these things is survey experts, and the results aren't too comforting: http://www.nickbostrom.com/papers/survey.pdf

>We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.

"Andrew Ng (I believe) compared worrying about evil AI to worrying about overpopulation on Mars."

Berkeley AI prof Stuart Russell's response goes something like: Let's say that in the same way Silicon Valley companies are pouring money in to advancing AI, the nations of the world were pouring money in to sending people to Mars. But the world's nations aren't spending any money on critical questions like what people are going to eat & breathe once they get there.

Or if you look at global warming, it would have been nice if people realized it was going to be a problem and started working on it much earlier than we did.

I don't know about the superintelligence risk. As a line of reasoning, sounds way too abstract at the present time. But what about the very predictable and obvious risk detrmined by the end of jobs? That is scary. Are there any analyses of the impact of that on society as a whole? It's not just mass unemployment - that has a different dynamic when it's temporary and in response to a contingent downturn. We talk about the end of jobs for good. How will that work out?

Obviously no one can prove that AIs would be "inherently good" because there's no definition of "good" that everyone agrees on.

I'd be more impressed by a Human Intelligence Project - augmenting predictive power to encourage humans to stop doing stupid, self-destructive shit, and moving towards long-term glory and away from trivial individual short-term greed as a primary motivation.

AI is a non-issue compared to the bear pit of national and international politics and economics.

So the AI Panic looks like psychological projection to me. It's easier to mistrust the potential of machines than to accept that we're infinitely more capable of evil than any machine is today - and that's likely to stay true for decades, if not forever.

The corollary is that AI is far more likely to become a problem if it's driven by the same motivations as politics and economics. I see that as more of a worry than the possibility some unstoppable supermachine is going to "decide" it wants to use Earth as a paperclip factory, or that Siri is going to go rogue and rickroll everyone on the planet.

Job-destroying automation and algorithmic/economic herding of humans is the first wave of this. It's already been happening for centuries. But it could, clearly, get a lot worse if the future isn't designed intelligently.

Being worried about "evil AI" is a lot like being worried about evil flying cars. Except people have built actual flying cars in the corporeal world, where "AI" is just a collection of signal processing techniques that don't work very well most of the time.

But hey, I labor in this domain: if paranoid richy-rich types want to throw money at it to ensure that they remain at the top of the heap, I'm all for it.

> But data data data, that could turn out to be very valuable.

Yes, but data also can be collected openly collectively, in the spirit of Wikipedia or OpenStreetMaps etc.

What I think OpenAI should encourage is the development of algorithms that can be used to crowdsource AI. I don't think there are good algorithms yet for model merging, but I would be gladly proven wrong.

> The "evil" AI and safety stuff is just science fiction

There already exist drones that kill based on AI.

Reply on news.ycombinator.com