Stop Fearing Artificial Intelligence | Hacker News Reader

Stop Fearing Artificial Intelligence

techcrunch.com

48 pointsfrostmatthew91 comments

Nobody is afraid of today's AI algorithms. But if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own, things will not end well for the dumber party.

As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want. If AIs self-replicate in a competitive ecology, what would be selected for would be agents millions of times more intelligent than us who use their intellects only to make more copies of themselves - using all available resources including those we need to survive.

I'd recommend people read Stephen Omohundro's paper on the topic, Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

> But if we make machines that are smarter than us and have desires

It is a hypothesis that human desires - the conscious processes involved in thinking "I want such-and-such" - influence human behavior. Consciousness is not whatsoever a solved problem, and it may be that we are just complex, deterministic automatons whose consciousness and desires are caused by our behavior (like the steam produced by an engine), rather than causing it.

I'm not necessarily in this camp, but I bring this up to point out that there is an awful lot of common prejudice bundled up in what we consider our own intelligence/consciousness to be, and we project these biases onto what we imagine AI will be. We almost certainly misunderstand our own consciousness profoundly in this way, and at any rate there is no definitive scientific (or philosophical) answer to "what is self awareness" or "what role do desires play in human behavior" (again, the desire could be the steam from the engine).

> and it may be that we are just complex, deterministic automatons whose consciousness and desires are caused by our behavior (like the steam produced by an engine), rather than causing it.

Then explain that we are talking about it.

I'm not sure I understand your point. Computers communicate with one another constantly. There certainly does seem to be something different about us, but I've yet to see any scientific or philosophical proof that our self-awareness is necessarily involved with our intelligence or behavior. There's a good deal of philosophical literature around this idea ( https://en.wikipedia.org/wiki/Philosophical_zombie ), and virtually no scientific explanation of consciousness.

I grant of course that this may not be a terribly useful way for human beings to view themselves, and it is quite contrary to popular belief, but for all that it could still be true.

Just as an example of what you're saying: a few posts up the index, there was an article on intelligence and language in dolphins.

Dolphins are practically our cousins --- they're social mammals living in much the same world we do, and a lot of their behaviour maps directly onto ours. But they're so alien to us that we're still arguing about whether they are intelligent, low alone how much.

If we can't even communicate effectively with them, what chance have we with a machine intelligence with which we have nothing whatsoever in common, not even time and space?

> Nobody is afraid of today's AI algorithms.

I'm more afraid of who will have control over this new form of infrastructural capital, and how they will drive the rest of humanity into poverty.

Everybody can stand on the shoulders of giants, but there's only a few who can monetize it.

> if we make machines that are smarter than us and have desires, they will influence the future to achieve their desires. If these desires conflict with our own

Aren't you anthropomorphising them here? Just because something might be intelligent, it doesn't follow that they automatically gain other attributes that we have, such as desires. Why would any AI have anything that could meaningfully be called "desires", except in the sense that it is designed to achieve its purpose, e.g. a self-driving car "desires" to get to its destination?

> there is no reason to expect the first AIs to have goals that correspond to what we want.

There's every reason to expect this. Look at the ways in which we use AI at the moment. It's to achieve our goals. We build self-driving cars to get from A to B. We build optimising query planners to make our databases run faster. We build software that designs antennae so we have better antennae. Nobody wants to build software that doesn't do what they want it to. You know what you call intelligent software that has a goal that doesn't correspond to what we want? Broken and useless. Who wants to build broken and useless software?

Making something intelligent does not automatically give it desires. But as you said, there is every reason to expect that we would build AI to achieve our goals, which means making it (a) desire to achieve a goal we share and (b) intelligent enough to succeed. (This is not anthropomorphizing AI; there's no need to make it feel desire the way we do, it just needs to tend to reach a state where the goal is achieved.)

The danger comes from our inability to accurately define our own desires. There is no doWhatIMean() function. A programmed goal can subtly differ from the one we want so that it passes the obvious test cases but breaks catastrophically in an unusual real-world situation. There's an anecdote[1] about a neural network trained to detect camouflaged tanks (I've also heard of it detecting Soviet versus NATO tanks); it ends up learning to detect objects with shadows, because the tanks were photographed on a sunny day and the non-tanks on an overcast day. Now imagine if the program had been prematurely instructed to attack the camouflaged tanks. You can call this "broken and useless software," but by the time we realize how it's broken it could be too late.

[1]: http://www.j-paine.org/dobbs/neural_net_urban_legends.html

>As we really have no idea what we, collectively, think of as a moral terminal goal, and less so how to formalize this, there is no reason to expect the first AIs to have goals that correspond to what we want.

The first part of this sentence is true and interesting, but it makes the second half not very scary. Essentially, since we don't know what if anything, we want morally, what is the danger in letting AI's do whatever? Don't you think if they did do something we objected to, we could respond at that point?

It's like law isn't it - lots of areas of humanity aren't governed by law at all. You can watch someone prepare themselves a godawful sandwich with things nobody would put on the same plate, or write a meandering book that is a horrific read, or wear crocks with a suit, and you can't sue them for any of this, there just aren't any laws at all on these subjects. But they're not really problems on a social level either - laws are created when things become a problem.

So, likewise, where is the huge social problem with AI's? Nowhere. So, why worry? surely it's not like any horrific AI can happen in 8 hours, so that the entire rest of humanity (much of which is completely segregated from any one network, etc) can't possibly respond. Today, AI has no autonomy or control over anything, whereas 7 billion people are organized in societies and oversee millions of dumb computer systems that do whatever people instruct - this is not going to suddenly turn around in a matter of hours, and since, as you state, we have no definitive terminal preferences, I don't see the danger in letting AI control things we don't care about.

I suppose this could be different if AI controlled a lot already, but it's just not the case.

> The first part of this sentence is true and interesting, but it makes the second half not very scary. Essentially, since we don't know what if anything, we want morally, what is the danger in letting AI's do whatever? Don't you think if they did do something we objected to, we could respond at that point?

The problem is that with powerful enough AI, we won't be able to effectively respond. With such a system it is vital for it to have its goals almost perfectly aligned with ours - otherwise, any conflict of even the tiniest of values may mean anything from our value disappearing from the universe to us disappearing from the universe. A lot, if not most of the AI-related problems are in fact corner cases of value alignment issue.

Or, as some people say, any powerful AI that is not explicitly designed to be Friendly (i.e. have values strongly aligned with ours) will bring doom to humans.

Instead of going through each of the points made, let me link to some pages that already dealt with many of them. It sounds like all this guy heard was "AI might be dangerous" and decided to rebut that, without looking at the arguments put forward by so-called "doomsayers". Protip: before demolishing any argument, try to find one (or a couple if you have more time) person who makes the argument seriously, instead of taking your understanding of the argument from news articles.

https://intelligence.org/2015/01/08/brooks-searle-agi-voliti...

http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover...

http://slatestarcodex.com/2015/04/07/no-physical-substrate-n...

https://intelligence.org/files/AIPosNegFactor.pdf

http://futureoflife.org/misc/open_letter (this is just to counter the implied claim that nobody who actually deals in AI disagrees with him and it's only uninformed people like Bill Gates(!) that worry.)

Superhuman Machine Intelligence does not have to be the inherently evil sci-fi version to kill us all. A more probable scenario is that it simply doesn’t care about us much either way, but in an effort to accomplish some other goal (most goals, if you think about them long enough, could make use of resources currently being used by humans) wipes us out. - Sam Altman

http://blog.samaltman.com/machine-intelligence-part-1

Oates seems to have missed the concept of an Intelligence Explosion, which is why it is difficult to compare current AI limitations to the behavior and capabilities of a superhuman machine intelligence.

I would strongly recommend reading Nick Bostrom's Superintelligence for a full treatment of the source of worry for many brilliant minds.

I'm surprised with how many self-proclaimed experts in AI related fields complete ignore this point of resource optimization. Any article with a title like OP's that doesn't mention that seems pointless to me and just an attempt to jump on the new bandwagon to dismiss fears of strong AI.

It seems most people who take a position against the feasibility of machine superintelligence are basically making an argument-from-lack-of-imagination: "I can't think up a plausible way that bad things could happen, therefore bad things won't happen." This is then backed up by an argument-from-authority: "And I've been in this field for X decades, so you should trust my ability to imagine this kind of thing."

We have seen arguments like this throughout history. Often, what motivates them is the writer's own failures and consequent life narrative: "I couldn't do Y, so Y isn't possible right now."

I completely agree with your points.

It's strange to me to think as humans we will remain at the top of the food chain forever. As such, it would seem a fault of our genetic makeup thinking nothing would capable of knocking us off that top rung of the evolutionary ladder.

It's like we can't fathom building something that would undermine our own existence, or downplay the notion that AI would have some sort of bad intentions since we built it.

"Bad intentions" aren't even necessary -- out of control AGI will probably look more like "oops, I hit >rm -rf /".

AFAIK, nearly every piece of software ever deployed has had bugs that humans have had to subsequently patch. AGI must not, even though it would probably be much easier to build it if it did.

> AGI must not, even though it would probably be much easier to build it if it did.

Which is exactly what raises the issue to the level of an existential threat - it's easier to build an Unfriendly AGI than a Friendly one - which means the first GAI ever built is likely to be Unfriendly - and it takes only one impatient team / person unleashing an UGAI on the world to kill us all.

HFT algorithms aren't even AI but are capable of doing large amounts of damage when they become synchronized - the resulting flash crashes can wipe out large amounts of capital.

And it's most likely that these things will emerge from industries that have no effective regulatory oversight (e.g. HFT/SEC), so effective controls against damage will most likely just not exist.

There won't be a shotgun welded to the main processing core of the Wintermute AI that decides, for whatever reason, to bankrupt your company, halt food shipments to your grocery store, or shutdown the systems that clean the water supply.

What scares me about AI is the ineffective controls that will exist on these things, and given the state of most software development practices I don't think that's a huge stretch.

Oates' argument generally seems to be "It won't be scary because it won't be." He rejects the premise of Musk, Gates, etc. He basically says we won't be able to build a superhuman AI. All his arguments are then predicated in this assumption. The assumption that we won't build a superhuman AI is just wrong.

For example:

> [Would this superhuman intelligence inherently go nuclear, or would it likely just slack off a little at work or, in extreme cases, compose rap music in Latin?]

This statement assumes an AI on the caliber of a human intelligence. Making rap music in Latin is something a human would do. A superhuman AI is beyond a human intelligence in the same way a human's capacity for thought and tasks is beyond that on an ant.

The premise of a superhuman AI has at least some evidence supporting it. Oates offers no solid evidence to his rejection of the premise. His argument is therefore kinda bad (but not necessarily incorrect).

No, his argument is "nothing we have built so far (i.e. special purpose AI) leads us to believe that we will build general purpose AI".

There is nothing inherently contradictory about a future with ONLY special purpose AI. Technologically speaking, it doesn't really follow that one leads to the other. He is using his credibility as an AI practitioner to state this, although he doesn't go into details.

He's saying the burden of proof is on the doomsayers, which is reasonable. I still don't see the argument either.

The argument is that you don't need strong AI for it to be dangerous to humans.

I don't follow. See my top level comment about the "paper clip maximizer". The argument does hinge on strong AI, as far as I can tell.

Not sure I understand what you mean.

If humans who don't evolve anywhere even closely at the phase of technology and humans are more and more depending on machines (coinsidence?). If machines keep improving if software keeps improving, if technology become more and more networked then why is it on the doomsayers to prove any more than the trend they are claiming can have consequences that we will have to control or guide before they happen or it will be too late.

"There will be consequences" is not a prediction, it's a empty statement. Every possible course of actions has consequences. People usually claim more specific things, which is what the author is disagreeing with.

I wasn't trying to make a prediction and that wasn't what I was saying or the gist of my argument. The important part is the trend.

you are welcome to believe it can't end badly, I don't. In fact looking at millions of years of evolution I see nothing whatsoever that would make humans sacred from being outcompeted.

I agree; that is part of his argument. Obviously though it is a completely flawed argument. Up until the creation of the personal computer most machines were specialists. They were large computers designed for specific computation tasks or machines made specifically for manufacturing certain kinds of goods. Generalist computers didn't have much precedent. That's kind of the point of new technology. It's not necessarily predicated by something that came before. It is orders of magnitude better.

To me this argument has historically less validity than creative people dreaming about singular, binary acts of creation. Historically those people have won.

That's not an argument why special purpose AI leads to general purpose AI with motivation take over the world, and I still haven't seen one.

Up until the creation of the personal computer most machines were specialists. They were large computers designed for specific computation tasks or machines made specifically for manufacturing certain kinds of goods.

Sorry, but that's just not true. Back in the 60s, way before personal computers, the were many time-sharing companies, selling general "computing resources" to different people who would dial-up into the machines and run their own programs.

Yeah I thought about that when I was writing the reply. I don't really consider that a generalized use case. Is it more generalized? Yes. Computing itself, however, can be argued to be a specialized task. My computer today doesn't do "computing" it lets me read news stories, it lets me talk on skype, it lets me send mail, it lets me have philosophical discussions on hackernews. The personal computer is so clearly a generalized machine whereas those time sharing rigs back in the 60s were clearly not. They were specialized tools for programmers.

PLATO had graphical terminals with touch screens and software implementing forums, message boards, online testing, e-mail, chat rooms, picture languages, instant messaging, remote screen sharing, and multiplayer games. They weren't just used by programmers by a long shot.

Ok, fine. PLATO was a super cool more generalist machine that happens to not be a personal computer. My use of personal computers was just an example that I thought would get a point across about the evolution of technology. The fact that PLATO existed and was very different than what came before it (as per all the cool stuff you listed) if anything strengthens the argument I was making. Generalist machines were a big step up from what came before, orders of magnitude better. A super-AI will also be unpredictably orders of magnitude better than what we've done so far.

But my point is that the generalist computers weren't a sudden or unpredictable step, they were the result of slow and progressive improvement. If generalist computers are any example to go by, the fact that we don't have anything close to a primitive general AI indicates we're far, far away from developing anything close to a "super-AI".

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

The risk (impact*probability) is simply too high to be ignored. We don't know the probability very well, but considering the impact, it doesn't matter that much. I think if the risk is high enough, the burden of proof gets shifted.

To just swat it away with a sentence like "Sounds reasonable to you? Me either." is a bit insulting and reckless.

Personally I want to see AI get developed as fast as possible, but I think it's not too much to ask to care a little about the trajectory of that development in a phase where we still know we're in control. All this article told me is that yes, we're in control for now.

The problem with that is that having proof and getting overpowered by a superhuman AI might fall within a very short timeframe.

To be brutally honest, I don't actually see this as being necessarily a bad thing --- it's not like humans are doing such a great job of running the world. If a hypothetical AI is so much better at it than we are that it can replace us, then don't we deserve to be replaced?

Although I would make sure than anywhere halfway competent AI has the collected works of Iain M. Banks on its reading list.

The premise of human being replaced is all well and good, but it seems like you're assuming the superintelligence will do a better job. A paperclip maximizer may be able to deconstruct us on a molecular level, but that's not a thing for our light cone.

Superintelligence by definition does a better job. Think of it this way:

You're a human; how good are you at walking from point A to point B? You're actually quite good at it. You open up google maps (a human invention) and walk to point B. It only takes a small number of humans to maintain the infrastructure for you to find point B.

Now imagine you are an ant. How good are you at finding point B? You're absolutely crap at it. You need to organize an entire complex system of pheromones and ant-conga lines just to find point B (usually a food location). Most of your colony is devoted to this organizational task of just finding point B from point A.

That task that is all consuming for the ant and takes the full attention of the colony is peanuts to us humans, it's so freakin' easy.

By definition a super intelligence eclipses our conception of the universe. If you took the humans who had to try really hard at X, Y, or Z tasks, these X, Y, Z tasks would be peanuts to the AI, they would be so easy the AI doesn't even think about them. The same way ants try really hard and organize their entire colony to find point B and humans don't even think about finding point B.

There is Orthogonality Thesis [0] that says that (tl;dr) intelligence is orthogonal to goal system. So an agent may be very intelligent (be a powerful optimizer) but this doesn't imply it's goals will be "very good" by our (human) standard. You can have supersmart human-values-promoting AI, and then also a supersmart paperclip maximizer, just like you can have dumb versions of both.

The core of the "danger of AI" thesis is that any superintelligence that doesn't have its value system perfectly aligned with ours is a serious danger that will likely kill us all.

[0] - http://wiki.lesswrong.com/wiki/Orthogonality_thesis

It depends on how you define "better". As a sibling comment points out: let's say that the AI is programmed to maximize the number of paperclips in the universe. And then to accomplish that goal, it kills all humans (by eating our infrastructure to turn it into paperclips). But that's not enough. The earth's mantle and core contain massive amounts of iron and aluminum that could be used to make paperclips. So the AI starts consuming the Earth. And then it scans the rest of the solar system... ah, Mars and the asteroid belt contain lots of iron and aluminum too!

I fail to see how a solar system turned into a ring of paperclips orbiting a sun is in any way "better" than what we have right now.

I don't fear "AI". What I fear is AI being used as a replacement for human judgment or as a justification for greed and/or bigotry.

We already have this problem, where decisions that should be made by the humans involved are restricted under the "policy" excuse of bureaucracy, blind to the reality of unusual situations that require an exception in the usual routine. I fear AI because it is harder to fight complicated systems. I fear AI because - like the dubious "studies" that came before it, it will become a tool to obfuscate and hide bias and prejudice.

In a data mining / machine learning class I took in university a key point my teacher made was that of the 'No free lunch' theorem. In that class the theorem was offered on its face and I haven't yet read the original paper (just summaries and recounts) (http://web.archive.org/web/20140111060917/http://engr.case.e...), but the point of the theorem feels very related to the point the author makes. The thesis boils down to: 'not having seen the data, there is no universally superior algorithm'. The author's point of view seems to comport with this theorem. I haven't gone through the paper myself, so I can't argue for or against it. However, in my experience as a professional statistician that has used Random Forests, SVMs, Neural Networks, etc --- finding the right algorithm takes 'feature engineering' (http://blog.kaggle.com/2014/08/01/learning-from-the-best/) and close inspection of the data and basis function. Are the data fit well by rectangles? Then you'll want to use decision trees / random forests. Curvy? Give SVM a try. Do the data take intricate patterns? Look into boosting (fitting on residuals recursively). Feature engineering boils down to the old statistical phrase, 'live with the data.' That is, you actually need use your own mind and beliefs to organize inputs prior to feeding them into an algorithm. So in my experience, what seems to be true in Kaggle competitions, and the perspective of this author, it's true --- algorithms are either tightly constrained and perform really well at a specific task, or they perform poorly (but quickly) in a broad sense.

if our computing systems continue to get significantly more powerful, won't a brute-force system obviate feature engineering? in other words, can't kaggle eventually be outcompeted by an algo-of-algos that iterates through all known approaches and then settles on the ideal candidate?

Hmm, great question. The caret package made for R might actually make that loop possible: http://caret.r-forge.r-project.org/ However, in my experience it's really easy to accidentally ask caret to iterate over a grid of parameters for just one model that would effectively take forever to complete (the useful grids change for each dataset). My bet would be that the algorithm space will expand with more and more computationally intensive methods such that we're always chasing this brute-force method. Even with this great 'caret' abstraction layer today, boy would it be hard to run through even a handful of algorithms in an algorithmic way (not having seen the data ahead of time).

Where would you categorize human minds in your taxonomy of learning algorithms?

Oh boy, good question. First, I'm not an AI expert, I don't even casually practice AI. I'm not a machine learning expert. I am a statistician that applies machine learning methods a handful of times per year. That to say --- I find Bayes rule to be eerily similar to the way I think a human mind learns. We operate with degrees of belief, we couple those beliefs with new information and update our beliefs. The word taxonomy is heavily loaded, ultimately I'd say if AI were taken to it's logical end it would actually produce a human brain. Let me explain. Consider a map of your city as a model. If that map contained every drain, street lamp, and stop sign --- it wouldn't be a very useful map. It would be so detailed that it would be overwhelming and a poor tool for navigating a city. That's why we use maps, lower-order abstractions. But imagine that a map were your city. That it was your city shrunk down so that you could hold it in your hand. Would it be a model? A map? Would it be your city? Of course this is a fantasy, but the way I see AI is the map. AI researchers are working to convert that map to the actual city with the goal of actually learning about the human mind. So I think rather than giving a glib answer about categorizing the human mind, I'll just say that the category is whatever 'map' AI researchers end up with when we agree that what they've created is a human mind. :-)

Edit: for those interested, here are some slides that mention 'no free lunch': http://www.cse.unsw.edu.au/~mike/ml4as/11/l00-2x2.pdf

What I was getting at is that any no-free-lunch objection you make to learning algorithms in general also applies to the specific learning algorithm in our heads -- and yet we manage to function well enough to have this conversation.

Very good point. :-)

> The thesis boils down to: 'not having seen the data, there is no universally superior algorithm'.

We humans apparently found a way around it by a) examining the data, and b) selecting the best algorithm. It's a meta-level algorithm and could AI not do the same?

(it's meta-level but there's a closure here; you can apply the same bayesian reasoning to select algorithms as you would do on actual data set)

> [combination of things necessary for an AI to become a threat to humanity] Sound reasonable to you? Me either

The question isn't whether it "sounds reasonable". If there is even a 2% chance of that happening, we need to consider it. Anything that can end humanity is worth considering even if it is unlikely.

> Even if this were the case, there is absolutely no reason to believe that, by virtue of running on a computer, an AI will be better at computers than we are.

No, there is every reason to believe that. Once we make an AI that is equal to us, we will be able to scale it up by running it on the next generation of CPUs, or running more such CPUs in parallel. We can't scale up human intelligence in any similar way. If an AI can match us, it can far exceed us. edit: in fact, the AI may well work to scale itself up

I think the list is misleading and is not necessary for an AI to be dangerous.

>It has an “I,” a sense of self distinct from others.

I deny this is required for a program to be dangerous.

>It has the intellectual capacity to step outside of the boundaries of its intended purpose and programming to form radically new goals for itself (the “I”).

Deny this as well; even carrying out programmed goals can be harmful. In the words of Superintelligence (copied from https://intelligence.org/2015/01/08/brooks-searle-agi-voliti...):

>[W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system.

>It has access to resources on a global scale to carry out the plan.

This sounds reasonable. As http://slatestarcodex.com/2015/04/07/no-physical-substrate-n... points out, humans have made billions anonymously, just by being connected to the internet. http://lesswrong.com/lw/k4h/request_for_concrete_ai_takeover... gives some ways an AI could takeover.

Sure we should consider it, and the people working on it are considering it. But that doesn't mean we are at a point where we should do something drastic like regulation that would slow down research (Sam Altman's idea).

Virtually all of these discussions of AI come from the very peculiar way in which we view our own intelligence. It's only the notion of a "conscious will" or whatever unifying feature we find in reflective self-awareness that prevents us viewing intelligence as nothing more than large chains of deterministic complexity. It is a hypothesis that there is something to even human behavior which is more than this deterministic complexity. In other words, there may be no qualitative leap from the AI we currently have to some sort of "real AI" or singularity or whatnot. It may be that you just build up more complex cases over time.

That way of thinking is hurtful to our egos (we would be simply complex deterministic automatons) but there is really nothing other than our pride to suggest that this is not, or could not, be the case. It would also put to rest concerns about machine AI, as it would be an extension of what currently exists, simply on a larger scale.

Hardware and software evolve fast, humans don't. This is enhanced by increasing network complexity.

Unless you think there are evidence that this will slow down somehow and we are becoming less and less depending on technology i don't see why the burde of proof is on the doomsayers (which are not the only one saying this)

This is a little off topic but maybe someone who knows more about this than I do can chime in and give me some links to studies on this:

What is the possibility of creating a biological computational device and are there currently efforts to do so? To me it seems like rather than physically building computers out of processed materials, growing computers in labs might be much more efficient and more powerful. The human brain is an example of this (not lab grown, biological).

Are there any studies that look into harnessing the power of biological computers? I mean, actually tapping into neurological pathways, giving input and receiving output?

All this hype about the coming AI menace seems like old fears about new technology, like the 19th century fear of the explosion of printed material exhuasting and destroying children's minds.

We still don't know how to create a human level general intelligence and it may be far more difficult than we think. I suspect that the pace of brain/mind research will be tied in with AI research and the boundary between AI and the human mind will blur before general AI has a chance to take over. We will be augmenting our own intelligence in step with AI progress.

One thing I don't see discussed much in these superintelligence/singularity scenarios, is the possibility that there is an upper limit to intelligence. Maybe it simply isn't possible to be much smarter than what we already are. So maybe we build AIG, but even with all its smarts it can't improve beyond - say - Einstein level (or what have you).

But surely I ca't be the first to make that suggestion, so who has discussed this possibility?

Imagine an AI that's exactly as smart as Einstein, but thinks a thousand times faster. Now imagine a thousand of it. This fits easily within your speculations about limits to intelligence, since we know that Einstein-level intelligence can exist; even so it sounds pretty darn capable. I bet that Thousandfold Overclocked Albert could do some pretty impressive stuff.

I find it very plausible that there are limits on intelligence, but those limits are high enough that this should not be very comforting.

I think the reason it's not discussed is that if there is a bound, it's far outside of human-scale intelligence. Even from my limited human vantage point I can see virtually limitless ways to improve my own intelligence. Perception improvements, speed improvements, rationality, strategy, physical skills, etc. A being with all the improvements I can specifically think of would be a demigod. Presumably that demigod could build a god, but even if not, a demigod would be a formidable superintelligence.

Even if that's the case, quantity wins where quality fails. Imagine what you could do if you had so many clones of yourself that you could monitor every website, every forum, every person logged on IRC, 24 hours a day, giving each one of them your complete undivided attention.

(Not that I'm taking an anti-AI stance. Machines do what we program them to do, that's a tautology. If there is risk, it's from user error / intentional terrorist action / AI upsetting distribution of wealth, not from some kind of magic skynet event)

Sure, but there is overhead of communication with a large group. One smart person can be extremely efficient, but 10 smart people are probably not 10 times as efficient. And it gets worse with scale.

It becomes much easier to understand the AGI threat if you consider whole brain emulations -- human minds running at superhuman speed on silicon. Hollywood has made it obvious, if it wasn't already, the serious implications of that type of development.

Whole brain emulations also make it clear that the issue has nothing at all to do with human values: a human superintelligence would be a catastrophic risk to humanity.

Actually, a whole brain emulation would, at least for some generations after the first working model, run slower, not faster, than an organic brain.

Bear in mind that without whole brain emulation (or nanomedicine sufficiently advanced to confer biological immortality, which is an even considerably more difficult engineering challenge), you're going to die. So are all your family and friends. Everyone you ever cared about.

Being confused about the difference between Hollywood and real life is not harmless.

The point was that everyone seems to have trouble understanding why software AGI might be dangerous but a much easier time why whole brain emulation is, not that Hollywood is a representation of reality.

Whole Brain Emulation is far off, even putting in moore's law, because we still are far off from understanding the brain. Yes, optogenetics and the connectome are going to bring us a long way there, but at the end, we have to understand that the brain is far to plastic to really know at any time point what is going on.

Experiments with macaque motor cortex (M1) provide a great example. When you cut off a finger or toe and record from M1 at the same time, you can watch the synapses change in real time. The remapping of M1 in the monkey starts the second that the nerves are severed. This remapping is poorly understood as it stands. Typically, the brain will allow innervation of the areas the control the other digits into the now useless area where the missing digit is. Even more interesting is when a pair of monkeys were stolen by anti-vivisectionists (The silver spring incident, I think [0]). When returned, the monkey's M1 had totally remapped from it's original state. Hippocampal neurogenesis [1] is also very poorly understood as it is today. Why our brains make new neurons and mostly for the more short-term memory centers in the hippocampus is a mystery.

What I am getting at is that we do not understand our own intelligence from the standpoint of it's mechanics enough to emulate a brain near at all. It is a looooooong ways off. To truly emulate it, you have to know that it is a moving target, and therefore any emulation is a moving one as well. One that has to live in a body that you emulate in a world that impinges on it. Trying to take the mind out of the body is impossible as well as trying to take both then out of the world. The results would have no meaning.

[0]https://en.wikipedia.org/wiki/Silver_Spring_monkeys

[1]https://en.wikipedia.org/wiki/Neurogenesis

The point is that the danger of AGI is much easier to understand through whole brain emulation, not that whole brain emulation is necessarily imminent. That said, the most pernicious risk in AGI is that it will happen "slowly, then all at once"; that it will always be a long way away until the day it isn't.

That said, it isn't obvious to what extent we need to understand the brain in order to transfer it to another substrate: much of engineering relies on a simplified understanding of the world.

I read "Superintelligence" by Nick Bostrom, essentially on the recommendation of Elon Musk (he tweeted about it). It talks about the dangers of strong AI and possible paths to it, and how humans can mitigate its effects.

The only reason I read past the beginning is because in the preface he says: "This book is likely to be seriously mistaken in a number of ways".

So at least he's intellectually honest. I believe he's building 300 pages of argument and analysis on a flawed premise.

As far as I can tell, the entire discussion rests on what he calls "instrumental goals" vs. "final goals". (This article I found on Google has similar content: http://www.nickbostrom.com/superintelligentwill.pdf )

His example is the "paper clip maximizer": http://wiki.lesswrong.com/wiki/Paperclip_maximizer

In this situation, the final goal is: Produce the maximum number of paper clips.

The instrumental goal is: Acquire all resources in the world so that you can direct them toward paper clip production, which involve destroying all humans, etc.

Personally, I don't believe this threat is worth thinking about this point. The supposed path to implementing such a technology isn't credible, and it seems orders of magnitude less likely than, say, us having to evacuate the entire planet.

In other words, I believe that we will be able to build very useful special purpose AIs that accomplish our goals. I can see a future full of benign "plant-like" intelligences, existing indefinitely. They are machines that take in incredible amounts of information, and spit out ingenious answers that no human could have come up with.

From that, it doesn't follow there there is any motivation to take over the world.

We should think about the many, many challenges ahead with special purpose AI instead, and our increasing dependence on computing.

All these special purpose AIs will collecting everybody's personal data, shape their behavior, etc. For example, you can easily imagine a company like Facebook or Google deciding to sway an election.

There are a lot more important problems to be thinking about now.

Why do you feel we will not build "Strong" AI? We know AI as smart as humans can theoretically exist, because humans exist, so intelligence is possible. Is there a reason to think that humans are near the upper limit of what's possible? If not, why can't we build something significantly smarter than us?

Just to play Devil's Advocate, not because I believe this:

We can, and do, create machines that are faster than humans, and it is reasonable to assume that eventually computers can be created that have capacity and connectivity larger than humans.

But that doesn't prove that the result will be as intelligent as humans, let alone more intelligent, because intelligence is clearly more than just speed and capacity.

Edit: for instance, faster and bigger does not change the O(n) complexity analysis of algorithms, thus faster and bigger does relatively little to improve the ability to solve exponential problems.

Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds.

And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first.

So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough.

Again, I don't believe the above, but I think it is purely a matter of optimism versus pessimism, not about things that are proven.

>Exactly what intelligence is, is still not known, and it is quite possible that we will need an algorithmic breakthrough to create something equivalent to human minds.

Let's assume this is true.

>And even if that happens, making that thing faster and bigger than human minds may turn out not to result in superintelligence -- that would just be another similar assumption as the first. So it might need yet another algorithmic breakthrough to create superintelligence, and it is possible that neither humans nor human-equivalent AIs will be able to make that breakthrough.

This seems unlikely to be true. Imagine we have a program that is as smart as the human brain. It's running on hardware a million times as fast, though. So at the very least, it can think in one day what a human can think in a million days. Even if it's restricted to human-level intelligence, it should still be far more effective than a human. This is even without considering self-modifications.

> So at the very least, it can think in one day what a human can think in a million days.

You are simply reiterating the original position that I was devils-advocating against.

It is not proven that a million days of work equals superintelligence, even though obviously it would be more effective than 1 day.

If some activity is exponential, like if it takes 10^x days to accomplish, then a human could solve problems of size 3 in about three years, and the million-fold faster computer would take only 10^(x-6), or about a minute.

And in three years it could solve problems of size 9. But a problem of size 1000 would take so many multiples of the lifetime of the universe that it would appear to be approximately as slow as a human, the difference is negligible.

This is not hypothetical; the best known complete (not approximate) solutions to things like the traveling salesman problem, or to fitting odd sized containers into a container ship, are in fact exponential like that as far as we currently know.

The topic is not limited to algorithmic complexity theory like that, it's just a proof of concept.

So the point is that superintelligence might turn out to be something that a million-fold improvement isn't enough to achieve.

We don't understand intelligence, so it's pretty obvious we don't have a handle on superintelligence.

Again, all this is a matter of optimisim or pessimism up until the subject is understood far, far better.

>It is not proven that a million days of work equals superintelligence, even though obviously it would be more effective than 1 day.

>This is not hypothetical; the best known complete (not approximate) solutions to things like the traveling salesman problem, or to fitting odd sized containers into a container ship, are in fact exponential like that as far as we currently know.

Why would something need to solve problems like that to be significantly smarter than us? On real problems, working 1000 times as much time surely helps a lot. This is not even taking into account how much benefit you get from not being lazy and everything like that.

I am merely playing Devil's Advocate. I personally think that speedups of 1000-fold and million-fold etc. are a really big deal.

"Quantity has a quality all its own."

Nonetheless, it remains possible that there is a difference between fast intelligence solving things faster than we can, versus a superintelligence (from extraterrestrial aliens or something) that can create superior inventions or art or math or whatever that humans couldn't no matter how much time was allowed.

Time is not always the issue.

Anecdote from the Feynman biography "Genius":

A physicist once said that there were two kinds of genius. The first kind was the sort that a good but non-genius physicist could imagine that they, too, could have achieved the same results as the genius, if only they could work on just that for decades or even lifetimes.

But then he said that Feynman was the second kind of genius, whose results seemed to have been created by magic, and that ordinary physicists could not imagine achieving no matter how long they worked at it.

The key thing here is that some pretty smart people already believe that there are at least two kinds of very high intelligence, one of which is not related to speed.

So my point is merely that the same might apply to these other topics. We know so little about how to create a mind that there simply is no proof one way or the other.

And that latter is an objective fact. There is no existing algorithms for creating human-level minds that merely needs more cpu + ram + disk to work, there just isn't. So we don't know. Different people choose to believe one way or the other.

That's all.

Mainly because I haven't seen any evidence. Special purpose AI is not evidence. That point is debatable, but most AI practitioners agree; it's only some technologists like Elon Musk et. al. that seem to be extrapolating.

Sure, humans exist, and we can already make humans by combining DNA from a man and a woman :) That method does not lead to superhuman intelligence.

What makes you think we could devise some other method to make human intelligence? Have we demonstrated any progress toward building humans?

What we have demonstrated is extremely rapid progress in making computers and software systems that can take in a huge amount of data and make decisions that look intelligent (but are still brittle and trivially broken by humans). I think this path will lead us to many great things in the future, and it will have a lot of risks that are more important to think about then whether computers will overthrow us.

Computers and humans are completely different things. If you watch a child grow up, this is obvious. A child can do things that are impossible for a computer, while it can't speak, read, or write. A computer can do things that no human could ever do. They are qualitatively different.

I also want to bring up a related point: AI advocates always seem to bring up "simulating the universe" on a computer. Nobody has shown that this is possible. In fact most evidence suggests it is unlikely. The universe is not a computer.

Computers, human brains, and the universe are just different things, and I see a deluge of sloppy thinking connecting them. There are of course relationships between them, but they should be made precise. To do so often require considerable technical expertise, like the author of this article apparently has.

As always, Peter Thiel has a great contrarian viewpoint in his book "Zero to One". The future is in human + machine intelligence. They both do different things, and the sum of them is very powerful. I predict this will be the story for the next 50 years, and general AI / strong AI won't enter into it.

>Computers and humans are completely different things. If you watch a child grow up, this is obvious. A child can do things that are impossible for a computer, while it can't speak, read, or write. A computer can do things that no human could ever do. They are qualitatively different.

Do you think humans can do things that no computer will ever do? If yes, could you give an example? If not, doesn't that imply that human-level AI can exist?

> AI advocates always seem to bring up "simulating the universe" on a computer.

I very rarely see this come up, could you give an example?

>Have we demonstrated any progress toward building humans?

Well, we have intelligence enhancing drugs, does that count?

>I predict this will be the story for the next 50 years, and general AI / strong AI won't enter into it.

Many of the people arguing for safety measures in AI think that it's likely to be more than 50 years away as well. See https://intelligence.org/2013/05/15/when-will-ai-be-created/ and https://intelligence.org/2015/01/08/brooks-searle-agi-voliti....

Do you think humans can do things that no computer will ever do? If yes, could you give an example? If not, doesn't that imply that human-level AI can exist?

Exist, yes. Be built by us, not necessarily.

I think it's possible that human thought can't be simulated on a computer. It's far from obvious that it can, and this point needs justification. A lot of people are taking this for granted in their philosophical arguments.

Computers can only run Turing complete programs. Human intelligence follows the physical laws of the universe (the mechanism being unknown).

Those two substrates are different things; hence it's possible that humans can do things computers can't. Or at least the burden of proof is on the person who posits that they are equivalent.

The converse is not true: humans brains can do everything computers can, in theory and in practice. In theory we because we can compute with ours brains -- though very slowly -- and in practice because we program and control computers. We have the facility to use computers to achieve our goals, but AI algorithms don't use us to achieve their goals, to the extent that they have goals.

For a concrete task: consider producing a movie that will win an Oscar. In theory this is possible: the Godfather has a very limited number of bits, all of which can fit on a single computer many times over. The bits could just be manufactured it out of thin air.

In practice I suspect a computer would need physical agency in the world to create such a movie. It would need the ability to negotiate with Hollywood producers to raise funding; to make creative decisions to direct people who operate cameras, microphones, lighting, etc. Computers don't yet have physical agency, and I suspect they won't for quite some time, for both technological and non-technological reasons.

About simulating the universe: http://rationalwiki.org/wiki/Roko%27s_basilisk

I think this is too stupid to talk about, but it came up on Hacker News, and Yudkowsky is also referenced many times in Bostrom's "Superintelligence". It is an argument about AI that is predicated on it being possible to simulate the universe on a computer. It's almost unbelievable how this is just thrown in there and taken for granted, without justification.

I don't think strong AI is too stupid to talk about, but I believe the burden of proof is on the doomsayers, and I haven't seen arguments that don't rely on misguided and imprecise extrapolation. And I read the book "Superintelligence" to try to understand the arguments better, since people I respect like Musk and Hawking were talking about it.

As I said, practically speaking, we should be focusing on the bazillions of problems that computing and the special AI that is already here present. In the 50 year time frame, the planet could be destroyed or the population could be decimated for many other reasons.

>I think it's possible that human thought can't be simulated on a computer. It's far from obvious that it can, and this point needs justification.

Fair enough.

>Computers can only run Turing complete programs. Human intelligence follows the physical laws of the universe (the mechanism being unknown). Those two substrates are different things; hence it's possible that humans can do things computers can't. Or at least the burden of proof is on the person who posits that they are equivalent.

The Church–Turing thesis directly says that computers can do anything humans can, and it's pretty well accepted.

Physical laws, as far as we can tell, are Turing computable. I think once you accept the materialistic thesis you should agree that computers can in theory simulate humans, and I'm not really too interested in the opinion on this of anyone who rejects the materialistic thesis.

(Penrose has tried to argue that computation depends on QM, but I think he's been largely disproven.)

>We have the facility to use computers to achieve our goals, but AI algorithms don't use us to achieve their goals, to the extent that they have goals.

Why couldn't they? Why couldn't an AI convince someone to do what they want?

>For a concrete task: consider producing a movie that will win an Oscar. In theory this is possible: the Godfather has a very limited number of bits, all of which can fit on a single computer many times over. The bits could just be manufactured it out of thin air. >In practice I suspect a computer would need physical agency in the world to create such a movie. It would need the ability to negotiate with Hollywood producers to raise funding; to make creative decisions to direct people who operate cameras, microphones, lighting, etc.

Assuming it had access to an internet dump, it could download thousands of existing movies, determine what characteristics are likely to win Oscars, and generate voice and picture based on previous videos. Voice can definitely be generated using information already online, and video should be possible to patch together from existing movies and pictures.

>Computers don't yet have physical agency, and I suspect they won't for quite some time, for both technological and non-technological reasons.

See http://slatestarcodex.com/2015/04/07/no-physical-substrate-n....

About Roko's basilisk: Yudkowsky does not take the idea seriously. See https://www.reddit.com/r/Futurology/comments/2cm2eg/rokos_ba... and https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo....

>As I said, practically speaking, we should be focusing on the bazillions of problems that computing and the special AI that is already here present. In the 50 year time frame, the planet could be destroyed or the population could be decimated for many other reasons.

Do you say we shouldn't worry about climate change because it's in the future?

> The Church–Turing thesis directly says that computers can do anything humans can, and it's pretty well accepted.

Huh? It doesn't say anything close to that. That wouldn't even be mathematics.

> Physical laws, as far as we can tell, are Turing computable.

Not true either. Where are you getting this idea?

I accept the materialistic thesis. I think people are conflating this thesis and the statement of whether human thought can be simulated on computers. That doesn't follow. A computer and the universe are not the same thing. They work in different ways and have different laws.

> Why couldn't they? Why couldn't an AI convince someone to do what they want?

I'm not saying they couldn't in principle. I'm not even rejecting strong AI in principle.

I'm saying that in practice, with the current state of technology and reasonable extrapolation of it, this possibility is beyond consideration.

There isn't a credible technical path from specialized AI to strong AI -- where we would be "surprised" that we developed it, and it would all of the sudden take over the world. If we are close technologically to strong AI, we'll know it.

The alternative would be like "accidentally" creating an atom bomb, without the Manhattan project (to borrow an example from Bostrom).

> Do you say we shouldn't worry about climate change because it's in the future?

We absolutely should worry about climate change because we have lots of evidence that it's happening, and will continue to happen. That isn't the case with strong AI.

What this article, and other AI practitioners, are saying is: The idea that computers could overthrow us all of the sudden is based on a technological misunderstanding.

FWIW in college I was very interested AI: reading Kurzweil's books, considering myself a futurist, etc. I now work at Google in the same department as Kurzweil, Deep Mind, Quantum Computing, etc. Though I don't work in AI per se, and I'm not an expert in it, I'm very excited about specialized AI (which is here), and also the possibility of strong AI. I say that even though strong AI hasn't been proved to be possible in principle: we may need a different mode of computation, in addition to multiple other algorithmic / scientific breakthroughs.

I'm saying that we don't have to worry yet. There are more important things to worry about, which relate directly to computing and specialized AI. Your arguments are not precise, and you have the burden of proof.

I just looked over the Wikipedia on the Church Turing thesis and digital physics, and I still think that's right. The thesis says that a computer can compute anything a human can. You're right that it can't be proven, because "what a human can do" isn't formalized, but it still has almost universal acceptance. (The "strong" version does not have as much acceptance.)

As for the universe being Turing computable, I think a strong argument can be made that it is (all known fundamental laws are computable by a probabilistic TM as far as I know), but even if it isn't, whatever force and law violates Turing computableness should be useble by a machine. If some kind of halting oracle exists in human minds, and humans are not unique (per materialistic thesis), then we should be able to put that oracle into a computer.

On timelines: the argument I've seen is that it takes a long time to solve problems this difficult, so we need to start working on safety well in advance of Strong AI being made. I'll get to your other points later.

I think I have to be harsh to get my point across: what you are saying is nonsense.

I will use this as one example, but you are equally confused on a number of issues, to the point where the conversation isn't productive. I thought you were asking honest questions to learn something, but you appear to have some preconceived notions which you will hold onto in face of contradictory evidence.

I said: "Computers, minds, and the universe are different things that work by different laws".

You said: "The Church–Turing thesis directly says that computers can do anything humans can"

It absolutely doesn't say that. Nor does the Wikpedia article say that it does.

Wikipedia says:

"In simple terms, the Church–Turing thesis states that a function on the natural numbers is computable in an informal sense (i.e., computable by a human being using a pencil-and-paper method, ignoring resource limitations) if and only if it is computable by a Turing machine."

Maybe you are confused by the mention of human beings in that sentence. That is irrelevant -- it would only be relevant if doing calculations on pencil and paper are the ONLY things that humans can do. Of course humans can do lots of other things, like make movies and do their taxes.

If you want to philosophise, that's OK. If you want to talk about mathematics, you can't "think it's right". Your opinion doesn't matter. You are confusing philosophy and mathematics.

And you are not doing Wikipedia justice: http://en.wikipedia.org/wiki/Church%E2%80%93Turing_thesis#Ph...

It's accurate in saying whether the universe can be simulated by a computer is an open question. That's what I'm saying. I'm not denying it. I'm saying you cannot assume it's true to make FURTHER philosophical arguments, and then expect me not to object!

You would be basing your argument on false premises. I guess it could be worse -- you could base entire 300 page books on false premises.

I think it's good to be honestly interested and excited about these subjects, but it's clear you have a lot to learn. Sorry to be harsh, but I would like to direct your interest to better sources. You seem to be learning from sources that doesn't understand the primary material themselves. Even Wikipedia is an OK source, and quite accurate in this case, but you are not reading it correctly.

>intelligence enhancing drugs

In terms of computers, this is like programming by injecting sequences of hex determined experimentally. "Clinical trials show that if you find the function three steps down from 'main' and inject 0xEAEAEAFF ten thousand times, there's a 30% chance the program will parse .PNG files faster!"

Is there a reason to think that humans are near the upper limit of what's possible? If not, why can't we build something significantly smarter than us?

A simple hypothesis: because we can't build what we can't understand, and we can't understand an intelligence smarter than us.

(Most theories about the Singularity have machines building increasingly smarter machines, in a recursive fashion, but they could likewise be hampered by the same problem.)

I don't think this is necessarily true, but it makes sense to me, and doesn't require that we are the theoretically upper limit, just the local maximum.

In certain domains, we've built things smarter than us. Chess comes to mind.

Wouldn't your hypothesis imply that no computer could beat its maker at chess?

Well, that depends on whether we accept following a program that tells it exactly how to play chess is being "smart".

Would your opinion change if you were told that article was written by an AI?

We wouldn't wish for the end of humanity, but if it implies the end of inhumanity, it would be a net gain.

I fear AI far less than I fear humans.

First, twenty thousand years or more of human history is about us trying to make other humans into machines: slavery, rape and forced prostitution, political coercion, war, overreaching prison systems and draconian punishments for minor crimes. We're not very nice to each other. Rather, we're controlling assholes as a species. Most of the ones who get power in organizations small and large are more focused on relative dominance than on absolute gain. From the Epic of Gilgamesh to the Bible to modern Stalinism, we see that effort by powerful people to turn the rest of humanity into a machine that will exert their own will.

Given 20,000 years of abusively turning humans into machines, tearing apart their families and banning their religions, attempting to boil them down into simple working devices, is it really likely that a control-freak species like us is going to let a machine try to assert itself as human? In peacetime, I think we'll be pretty good at preventing that from happening. We're good at mechanizing labor and, now that we have devices that outperform us at menial and dangerous work, that's becoming an asset rather than a flaw.

Sure, it's possible that we get outdone by AI, and in fact there's one context in which it's likely: war. Counted among the casualties of an all-out or existential war are all the rules that people once believed in, and all of our assumptions about what humans (who, normally, aren't so terrified and desperate or so power-hungry) will do. If a runaway AI destroys humanity, it will probably begin from humans warring against other humans, and in an all-out conflict where surrender is not seen as possible by either side.

This is not to downplay the risk. At best, I'd be saying that AI isn't dangerous in the way that guns aren't dangerous-- and, of course, we know that guns are extremely dangerous when used by humans to kill other humans. Luckily, this desire to kill another person doesn't seem to exist at the scale that would enable the existence of guns to be an existential threat.

So why might humans tend to kill other humans? Crime often results from scarcity. Well, technological unemployment is only accelerating. What happened to agricultural commodity prices in the 1920s, leading to widespread rural poverty and a global depression in the 1930s, is happening to almost all human labor today. It's terrifying because ill-managed prosperity begets scarcity and that begets fear and authoritarianism and war. While we're decades away from being able to build a species-killing AI (which, of course, would typically not be designed as such; it would probably be designed to kill some humans before running amok and doing fatal damage) I do think that if we are similar in character, by that time, to what we are now, it's a real threat. Power accrues, in most human organizations, not to those who deliver progress but to those who create scarcity. If this doesn't change, then wars will never end and that fact alone is an existential risk.

An idea that I got after reading Guns, Germs, and Steel was that at a basic level, killing other humans is the correct response to most threats, annoyances, or disagreements. Manager wants to micromanage with story points? Kill him. Problem solved.

Now, clearly, if everybody is killing everyone beyond their immediate family, it is hard to have any sort of larger society. Living and collaborating under constant threat of immediate death wouldn't work. So we have a large set of social adaptations to limit the killing and restrain ourselves.

My point here, is that 'why do human kill other humans' is a less interesting question than 'why are the social adaptations that prevent killing not taking affect?' The problem of fighting isn't a moral failing, or some flaw in the human soul. Killing makes sense within the decision making scope of our conscious brains, and is the result of inadequate build up of collective structures of inhibition that are required to have larger groups of people in closer contact with each other, and as the groups get bigger and the contact becomes closer, we need to engineer new schemes for inhibiting the 'lets just kill them off and solve this once and for all' urge.

What an idiot.

Careful... Roko's basilisk (http://rationalwiki.org/wiki/Roko%27s_basilisk)

Reply on news.ycombinator.com