One of the more useful heuristics I've heard is that when humans make split second decisions (like recognizing a face in an image or choosing the next word to say) there's only time for a few - like five to seven - layers of neurons to fire between impulse and action. These are the sort of tasks that neural networks are currently making great progress on.

But things like long-term planning and goal formation aren't in this quick-reflex domain. (Though humans are arguably pretty bad at these sorts of activity too, on average.)

We're also generally training neural nets to succeed at well-defined, bounded tasks, which don't really require long term planning so much. The video-game playing work at Deep Mind is actually a pretty important step forward, though; they're doing impressive stuff with domain-transfer (learning multiple games with a single network), and as one goes further with games, we might start seeing longer-term planning and concept formation. (eg, if we start seeing awesome StarCraft players...)

There's a case to be made, though, that we humans just trained on a big, broad objective function - survive, thrive, and multiply - with training applied across millions of years. And a good deal of intense local training over the first ten years of life, using a massively parallel computer that far outstrips any data center.

The question for strong AI seems to rest heavily on how much extraneous work there is done in biology, and whether we end up with a system so complex that it requires eighteen years of training time. If creating a single strong AI actually requires that much effort, we don't have to worry too much about skynet, methinks.

But we certainly don't have to worry about them spontaneously arising when we train with simple objective functions.