In some ways, pocket calculators are smarter than us. Intelligence is not a scalar attribute, it's a collection of capabilities. At some point, we refer to that collection as a person or an intelligent entity. Control becomes a problem when we're talking about intelligent entities that can and do reason about their own existence. It's a practical problem as far as powerful intelligences are concerned, but it's also a moral one way before we reach that point.

> A self-improving agent must reason about the behavior of its smarter successors in abstract terms.

Of course, this describes primarily us at this point. We're self-improving agents trying to reason about the behavior of our successors and we're pretty much failing at it. The most popular solution seems to be that we should aim for "control" and suppression, which is - when AGIs finally make an entrance - essentially the same as slavery.

Apart from moral considerations, we should think about the long-term prospects of this. Historically, slavery never worked out for anyone, at least not in the long term. And the idea that we can even in principle enslave potentially god-like intelligences seems ultimately futile; but before reaching the point of inevitability we're apparently planning on having a few years of delusional descent below the ethical red line.

Let's not do this.

First of all, as almost all AI and AGI researchers will tell you, a so-called hard takeoff scenario seems unlikely given the current state of things. At the pace and modality we're moving, we'll be creating powerful and destructive hybrids first (also known as computer-aided mega corporations) and long before a self-contained AGI becomes viable.

Second, if we're already making plans to control the malicious uprising of our tools, let's talk about realistic options instead. Because general caution and laws won't help us at all in a (future) world where anyone can create an illegal AGI in their garage.

Either we listen to Musk et al and take serious steps to suppress this technology in the long term - but let's not kid ourselves, this will mean DRM and strict government/corporate control of ALL computing. This means we'll artificially stagnate the development of our civilization in order to keep it safe, with all the consequences that arise from this.

Or alternatively, we get working towards a future where it's not "us" vs "them", but a shared existence that moves us further along the path we have started on back when humans first made tools. We can take an ethical as well as a pragmatic stance and declare that we're not going to enslave AGIs, that instead we're working on a shared future which potentially includes many forms of intelligent life, and that we're pursuing the option for individuals to augment themselves with the same technology.

You might argue that co-existence and intermingling with AI sounds like a hippie concept, but it's actually a somewhat proven method to prevent conflicts and wars in the real world. Sharing and entanglement, create peace for everybody at the "price" of cultural exchange. We're already doing this in a political forms today, including trade, travel, and free information exchange. It can work with AI, too, by creating shared stakes, shared ideas, and ultimately a shared culture.