The Road to Self-Reproducing Machines

There are ways to keep an AI in sealed hardware and making sure it can't affect the world, for instance by using an objective function that only deals with mathematics, and doesn't deal with the real world at all.

E.g. the AI is given a fixed amount of hardware and told to produce an algorithm that solves some NP-complete problem (say integer programming) in expected time as close to polytime as possible, as well as a mathematical proof that the algorithm satisfies the claimed close-to-polytime complexity bound. Then humanity can just solve the NP-complete problems separately once they have the algorithm.

This objective function doesn't care about the physical world -- it doesn't even know that a physical world exist -- and so it's about as likely to directly affect the physical world as MCTS or AlphaGo.

The "AI is going to run out of control" is a very compelling narrative (as everybody who has read the Sorcerer's Apprentice understands). But that doesn't make it true. Beware the availability heuristic.

(Incidentally, I think AI destroying mankind because it's too smart is an unlikely outcome. It's much easier for the AI to subvert the human-designed sensors linked to its objective function; and if the AI is sufficiently smart and the sensors aren't perfect, then it can always do so.)

So first I made a fire drill and start a fire, then dig out some clay and fire it into bricks... *some time later* ... then once the UV laser lithography unit is finished, it's time to assemble the 5nm chip fabrication module for the processing units and Voila. That's the whole thing finished!

I jest, but the thing is just because a machine is capable of self replication, that doesn't mean it's any good at making anything other than itself. Organisms are extremely good at making copies of themselves, but they're ultra-optimised for just that. Even the most sophisticated organisms are only able to make a small set of very crude intentional modifications to their environment, if any. We're the only exception.

Self replicators and omni-replicators are very different things.

Unless you’re suggesting the shirt is itself made of replicating units, I think you’re underestimating the gap between what Primative Technologies et al can do versus the foundation needed to make even a simple microchip.

And given life is self-replicaing nanotechnology, I also think you’re overestimating its limitations, in particular speed.

Your perspective on self replicating nano bots as well as some other comments I have read here paint a picture of immense change in a short time once the technology is released into the wild. This view seems to ignore the fact that the capability to reproduce and to adapt are not the only factors. Any green and grey goo has to cope with the limitations of it's surroundings. Here are some examples: * available energy sources (food, sunlight, wind, geothermal engery, ...) * the rate of energy conversion into the energy source consumed (electricity or sugar/fats, etc). * processing power (along with it's energy efficiency) * energy consumption/efficiency when interacting with the environment * speed of reproduction * speed of adaptability (through intelligence or evolutionary techniques).

I don't wish to downplay risks involved but even if someone creates an intelligence that can outsmart them or a replicating system which they fear may over replicate the creator should be in control of enough of the environmental factors to keep it under control long enough to reduce their creations access to energy, resources. The real risk (from a perspective of self preservation) is not limiting their creation enough or overlooking sources of energy, information or whatever needs to be limited for this system.

Nuclear proliferation is like our training wheels for all the dangerous techs that are to come. If you are worried about it, look into how world governance works, how international norms are agreed on, and take a partisan stance against the parties that block the ability to do such things.

Nuclear proliferation: so far so good.

Ozone layer destruction: disaster adverted

CO2 limitations: we would have solved that by now without GWB suddenly deciding that disbelieving in climate change was now a respectable policy position.

> The destruction that nuclear weapons can cause is nothing compared to the potential destruction from self-replicating machines

I think you overestimate the speed at which this will come. We probably could, today, make a self replicating machine from raw materials but it would be the size of a factory and would have a replication cycle of a few months. Going from self-replication to grey-goo will take some time, hopefully we can prepare to it.

Which is a known bug to over-idealize the unknown. Machines will uphold there programming, but are inflexible and will react wrong (shooting down rockets starting) in the rare situations (large asteroid heads for earth)..

Remote inspection and checksum of the device firmware can solve this problem. You should worry about intentional hacking of the firmware, i.e. self-replicating botnet.

In fact, all living things form a category of self replicating machines.

Depending, your [highly optimized] self-replicator might have some stiff competition.

I used to be concerned about this issue, due to the obvious analogy with biological evolution, but later I realized that it's an easy risk to guard against.

It's a straightforward engineering problem to reduce the possibility of accidental program mutation to any arbitrarily low level, for example using SHA-2 (with which the chance of an undetected error is 1e-77). Of course, you can have "somatic mutations" where one part of a machine or another malfunctions, for example due to damage or errors during construction; but those don't get propagated to the next generation, so they don't produce the kind of progressive deviation you're describing.

This doesn't happen in nature for a variety of reasons, among which is that as such corrective mechanisms become progressively more perfect, the evolution of the species using them becomes progressively slower, and therefore the perfection of the error-correction mechanisms never quite arrives. Moreover, any species whose evolution becomes very slow is at a major disadvantage when the environment changes sufficiently; it will probably die out and leave no descendants after the next climate change, even something minor like an Ice Age, much less a meteor strike.

Another thing to keep in mind is that the number of generations is quite limited in practice. If, to take an unrealistically risky example, you have an alcohol-dependent nanobot replicator weighing 2 picograms, and you set it to reshaping a tonne of alcohol-soaked soil, it can't make more than 5e17 copies of itself, for which it only needs 59 generations. We aren't talking about thousands or millions of generations: even if time allows for them, space doesn't.

1e-77 (.000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 01) is a number that it can be difficult to get a handle on. If we, to take another unrealistically risky example, converted Ceres into 2-picogram nanobot replicators, there would only be 4.7e35 of them, so the chance that one of them would have a mutation in its program that SHA-2 couldn't detect would be 4e-42, assuming that there were just as many erroneous replications as correct ones. (If only one out of every 1000 nanobot firmware installations had a copying error, the chance that one of them went undetected would instead be 4e-45.) The universe is only 4.3e26 nanoseconds old.

So, human malice or extreme carelessness would be necessary.

My understanding is that von Neumann recognized that science would produce these types of weapons eventually, and so pushed hard to reach an equilibrium state of non-use of nuclear weapons under Mutually Assured Destruction (the dude did found modern game theory). So I think it's a bit disingenuous to say that he tried to start nuclear war.

Certainly von Neumann is far from infallible, but his views on the safety of technology were based on reasoning and knowledge, not what would make exciting prime-time TV. We're literally contrasting the inventor of game theory and modern meteorology* with a TV show where sounds travel through outer space and all the extraterrestrials speak English.

This is like discussing whether Stephen Jay Gould or Jimmy Swaggart has a more credible opinion about evolution. I mean, punctuated equilibrium might or might not be correct, but you're being ridiculous.

https://whatever.scalzi.com/2009/10/13/teching-the-tech/ https://archive.is/khTlp

> when you admit that Star Trek has as much to do with plausibly extrapolated science as The A-Team has to do with a realistic look at the lives of military veterans, life gets easier. ... Meta to this is the discussion of why we have to accept that film/tv SF is riding the shortbus — there’s no actual reason it has to be that way — but let’s not get into that right at the moment.

______

* As well as, as you point out, a significant contributor to the development of the fission bomb, the hydrogen bomb, and modern computation. He was also the guy who axiomatized quantum mechanics and the foundations of mathematics, discovered continuous geometry and quantum logic and Hilbert spaces, and solved the compact-groups case of Hilbert's fifth problem. I guess you don't know much about mathematics, so that won't mean much to you; suffice it to say that the "von Neumann machine" was among the least of his achievements. Not bad for a chemical engineer.

We would be fools to disregard the thinking of great minds like John von Neumann, but even more fools to consider Gene Roddenberry a great mind. For the purposes of prognostication, he wasn't a mind at all; he wasn't even trying to predict what would happen in the future, but to dress up 19th-century Horatio Hornblower naval drama and 20th-century social commentary in whizbang clothing, so people would watch it and he could sell Chevrolets and vitamin pills. See https://news.ycombinator.com/item?id=28519184.

> Grey goo is not so much fiction as it is the logic consequence of the kind of machine its creator "designed" (on the sci-fi definition of designing, but with a very strong focus on science).

That's true; I think it's one of the reasons Zyvex's designs didn't include nanobot-sized constructors, instead preferring "convergent assembly" using nanometer-sized parts but not nanometer-sized replicators. Instead, they envisioned that the minimal self-replicating unit would be the size of a laser printer. A person armed with a rock could disable a warehouse full of such devices in a few hours if that became necessary.

So, self-replicating nanotechnology doesn't imply a risk of gray goo.

As a more useful point of reference than space opera and Drexler's thought-provoking speculations, Linux and GCC are capable of replicating themselves, but they don't pose the same risks as the Morris worm.

> that is a completely different threat model from any machine we have ever seen.

Even without the bogus gray-goo threat, that's true of self-replicating machines, but so were my other examples at the time. Well, maybe the PEPCON disaster wasn't so different from the items in https://en.wikipedia.org/wiki/List_of_ammonium_nitrate_disas... and https://en.wikipedia.org/wiki/Largest_artificial_non-nuclear..., in particular https://en.wikipedia.org/wiki/Wanggongchang_Explosion in 01626, which arguably brought about the end of the Ming 18 years later. Nothing like the Wanggongchang explosion had ever happened before, though, and nothing like it would happen again until the 20th century; quite possibly it was the largest human-made explosion before the US started the first nuclear war in 01945, though there are a couple of other candidates.

Oh, I've reread your comment, and you want progress on self-replicating.

RepRap won't lead to a full self-replicating machine. It's by design. The focus on self-replicating is because the most a machine is able to build itself, the cheapest its parts will be. A 3D printer nowadays can print almost every non-commodity part of itself that isn't electronics or runs very hot. There's no progress for it to make on that dimension. It has successfully taken down the price of printers.

What progress has been made on the last decades were about things like easiness of printing, printing speed, materials handling, printer size, print volume, quality of printing.