The brain “doubles up” by simultaneously making two memories of events

I have yet to hear a realistic strong AI nightmare scenario (i.e. one that doesn't presume a magically-smarter-than-all-of-us-combined AI as a McGuffin with no solid functional grounding) with a probability anywhere near offsetting that cost. Besides, stacking the constant chronic cost of a few tens of thousands of deaths against a nonzero-probability species-ending scenario is nearly an apples-to-oranges comparison---by the logic of simple probability-to-risk modeling, we should be stopping all disease research right now and pouring 100% of that money into fast asteroid detection and mitigation solutions.

Practically speaking, I think we have plenty of time while we are doing the research to cure a current and real disease to puzzle through the detection and mitigation strategies for a strong AI threat that is---at best---decades out (and, some would argue, will come to pass inevitably with or without our Alzheimer's research, yes?).

We have similar, when we optimize for "what works" (instead for e.g. "what sells").

The only existing instance of what we're trying to build - intelligence - is something a dumb, random, incremental optimization process following simple rules managed to somehow stumble upon. Now, if there's a strong local optimum in the design space of intelligent machines, then it seems plausible that evolution ended up there, and that we may stumble upon it too, thus converging with the evolutionary solution somehow.

Now I'm not saying our solution will be identical to biological brains. We have different goals (hell, we have goals, nature does not). But we're likely to end up doing many aspects of it in a way that resembles biology.

The core observation here is that it's the structure of reality (implications of laws of physics) that shape the search space we're traversing. Compare flight. Yes, human planes are very different from birds - but that's because they have less efficient energy sources, and also because we want them to go faster (have you ever seen a supersonic bird?). Still, both share some aspects, like the airfoil. Both we and nature "discovered" those because airfoils are dictated by the laws of physics - that's how you do flight in gases.

--

Basically, what I'm saying is that humans always describe things in terms of the technology of their age, but that doesn't mean it's wrong. Better technology means better description. Birds are unlike planes, but analyzing them using the model of airfoil we developed is a good idea and leads to more and better understanding.

--

EDIT

> Optimization seems to require some objective. In a practical sense, you cannot optimize a piece of software if you don't know what you are optimizing for.

Yes, optimization always has an objective - that's how we define it, in contrast to complete randomness. But the objective can be implicit or explicit. Explicit goals require a mind to be involved. Evolution has only implicit objectives, human-driven process have both (because we suck at knowing what we actually want).

But the second important part of an optimization process is the shape of the optimization space. This here is defined by laws of physics. And insofar as evolution's implicit objective is in some aspects similar to our objective, both get similarly influenced by the shape of the optimization space :).