If you haven't read Bostrom's book yet, I'd really recommend it. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
If you haven't read Bostrom's book yet, I'd really recommend it. http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
It's about as likely as a meteor hitting the planet and wiping out the entire human life. Possible? Sure. Should I be panicking about this right now? Nah.
The book is nonsense. I'll start paying attention when someone who has real experience in the field of AI research (and I'm not talking charlatans like Yudkowsky here), but someone like say Norvig come out and say it's a reasonable concern today.
Many computer science professors have publicly said they think AI poses significant risks. There's a list here: http://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-ri...
Also see this open letter, signed by hundreds of experts: http://futureoflife.org/ai-open-letter/
Some experts in the second link you gave are concerned, sure. But I can probably find an equal number who dismiss it as well. There isn't a clear consensus over AGI. I still remain skeptical. Same with your first link, which tries to "forecast" AGI. People can't forecast next month's weather correctly, so forgive me for not believing in a 10% chance in 10 years.
Actually, I take back my appeal to authority argument in its entirety, because I just remembered the first thing I saw in my AI class was a video of experts claiming the exact same thing. The video was from the 50s.
EDIT: Found it: https://www.youtube.com/watch?v=rlBjhD1oGQg
Also, no-one is worried about a skynet scenario. The worrying scenario is just any powerful system that optimises for something that's different from what humans want.
Second, the point is that even uncertainty is enough for action. For AI to not be a problem, you'd need to be very confident that it'll occur a long way in the future, and that there's nothing we can do until it's closer. As you've said, we don't have confidence in the timeline. We have large uncertainty. And that's more reason for action, especially research.
Consider analogously:
"We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Seems like a bad argument to me.
I'm 100% with Torvalds on this when he laughs at the prepostorous notion that AI will become a doomsday scenario. I think it'll become more and more specialised, branch out to other fields and become reasonably good. But there's a huge leap to go from there to HGI.
> any powerful system that optimises for something that's different from what humans want.
Except this notion rests on the premise that humans will not be in full control, which leads to the exponential growth argument which leads back to the Skynet like scenario.
> the point is that even uncertainty is enough for action
And that action is...what exactly? People won't stop building intelligent systems. There is no real path that we have from where we are to HGI, so it's not like researchers have a concrete path. Just what exactly does this research look like?
> "We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."
Extremely Poor analogy. We have decades worth of concrete data that tells us the nature and reality of climate change. We demonstrably know that it's a threat. Can you say the same about AI?
Also, I'll take the time to re-iterate how heavily skeptical I remain of groups like MIRI that are spearheaded by people who don't believe in the scientific method, believe in stuff like cryogenics, have history of trying to profit off of someone else's copyrighted material and have someone managed to convince a whole lot of people that donating them is the best way to fight off the AI doomsday scenario. People should do their research before linking to stuff like that. :(
In general, if there's a poorly understood but potentially very bad risk, then (a) more research to understand the risk is really high priority (b) if that research doesn't rule out the really bad scenario, we should try to do something to prevent it.
With AI, unfortunately waiting until the evidence that it's harmful is well established is not possible, because then it could be too late.
What AI risk research could involve is laid out in detail in the link.
I would guess A. the development of "known safe" high-level primitives, and B. a coupling of education in the craft of AI with education in the engineering of AI.
The "profession" of developing strong, general-purpose AI should basically look like a cross between cryptography and regular old capital-e-Engineering: like crypto, you'd constantly hear important things about "not rolling your own" low-level AI algorithms; and, like Engineering, the point of the job would be making sure the thing you're building is constructed so that it doesn't "exceed tolerances."
The research goals in the field of "friendly AI" are thus twofold:
• work in understanding possible computational models for minds, to understand what sort of tolerances there can be—what knobs and dials and initial conditions each type of mind comes with;
• and work in development of safe high-level primitives for constructing minds.
Both of these are the subjects of active papers. Note that these can both also be described as plain-old "AI research"; they're not just abstract philosophy, these papers are steps toward something people can build. The research within the subfield of "friendly" AI just has different criteria for what makes for a "promising avenue of research", e.g. ignoring black-box solutions because there are no knobs for humans to tweak.
> MIRI ... spearheaded by
MIRI is two things, and it's best to keep them mentally separate. It's a research organization, like Bell Labs—and it's a nonprofit foundation that funnels money into that research organization.
Yudkowsky is the director (head cheerleader) of the nonprofit. He doesn't really touch the research organization. The research org will succeed or fail on its merits (mostly whether it hires good researchers), but the leadership of the nonprofit has not-much to do with that success or failure, any more than AT&T had any impact on the success or failure of Bell Labs. You can believe in the research org known as MIRI even if you actively distrust Yudkowsky.
That's quite a strong claim - could you provide evidence for it? Are you referring to HPMOR? HPMOR has always been given away for free, both online and in print, and J.K. Rowling has explicitly allowed non-commercial Harry Potter fanfics.
The Singularity Institute for Artificial Intelligence, the nonprofit I work at, is currently running a Summer Challenge to tide us over until the Singularity Summit in October (Oct 15-16 in New York, ticket prices go up by $100 after September starts). The Summer Challenge grant will double up to $125,000 in donations, ends at the end of August, and is currently up to only $39,000 which is somewhat worrying. I hadn't meant to do anything like this, but:
I will release completed chapters at a pace of one every 6 days, or one every 5 days after the SIAI's Summer Challenge reaches $50,000, or one every 4 days after the Summer Challenge reaches $75,000, or one every 3 days if the Summer Challenge is completed. Remember, the Summer Challenge has until the end of August, after that the pace will be set. (Just some slight encouragement for donors reading this fic to get around to donating sooner rather than later.) A link to the Challenge and the Summit can be found in the profile page, or Google "summer singularity challenge" and "Singularity Summit" respectively.
Like AI based-HFT? Which humans want what? I'm pretty sure, that if there's one constant, that there always will be some humans on either side of the argument.
So Transcendence in leu of The Terminator. It's still hollywood fiction.
My point is that the entire idea of an buggy AI destroying humanity is absolutely divorced from the reality of AI research. It's not unimaginable to the lay-person, but there is simply no way of getting there from the neural networks we're making now.
To begin, can you go ahead and explain what self-awareness means in terms of a neural network you are training?
Here's a comparison: wouldn't it be great if we had started thinking about climate change way back at the beginning of the industrial revolution, before we decided to create tons of open-air coal plants?
There are reasons to solve problems before they're problems.
So is it worth exploring? Sure. But I'm not going to be concerned about something for which there is zero evidence to support it.
There are no reasons to solve problems before there is foolproof evidence that they are indeed problems.