In fact, it'll be worse: Humans are currently a net source of growth, but they'll switch to being a net drag on growth. So the decision-makers will be forced to sideline humans in order to compete.
Still, I wonder if any humans have been in control, really, since the agricultural revolution. Billionares building bug-out shelters? They seem even more scared than the rest of us.
Surely if we were really in control we could have come up with a better system than this.
Empowerment as 30% infant mortality rate.
Empowering life long marriage since a person would get married and be dead in 10-15 years.
Brilliant.
One of Western society's glaring cognitive dissonances: the conviction that "keeping people happy and healthy enough to work" is empowering them. Even assuming that the word "empowering" makes sense; even assuming that we can make sense of the notion of an authority "empowering" someone (which I personally cannot).
Directionally agree with rhelz but would push it further: any technique, even those which may have preceded agriculture, already does all the things you're claiming AI is going to do. Even a procedure entirely implemented by humans can keep its weighting of any unwanted form of "human input" beneath any epsilon.
------
> 2. There are effectively two ways these systems maintain their alignment: through explicit human actions (like voting and consumer choice), and implicitly through their reliance on human labor and cognition. The significance of the implicit alignment can be hard to recognize because we have never seen its absence.
> 3. If these systems become less reliant on human labor and cognition, that would also decrease the extent to which humans could explicitly or implicitly align them. As a result, these systems—and the outcomes they produce—might drift further from providing what humans want.
You talk about empowerment, but many of your arguments seem oriented toward alignment. Voting and consumer choice may indeed be techniques for aligning (and thereby binding and scaling) society (i.e., a given group of people), but they have very little ability to "empower" any given individual. The power of the individual voice literally decreases in proportion to the success of these techniques (i.e., in proportion to the growth of those groups of humans which compose them). In other words, your "explicit" techniques are alignment techniques and have little to do with empowerment.
Your "implicit" category (labor, cognition, etc.), on the other hand, does seem to me to be oriented toward something like individual power. Unlike voting and market-making, labor and cognition do seem to be (or can naively be viewed as being) oriented more toward something like our everyday notion of individual power than they are toward these notions of social "alignment" and top-down "techniques of empowerment." That is, without much mental gymnastics, we can imagine labor, cognition, etc., as coming from within the individual and radiating outward — which is probably as good a criterion of power (individuality, sentience, free will, subjectivity, ego, humanity, etc.) as we're ever going to get.
You seem to be claiming that AI is a relatively new threat to this category of "implicitly empowering forces." This is where you're going to lose the brighter minds in your audience. Because has there ever been a more dominant and monotonous trend in human society than the reduction of the dimensionality of human labor and cognition, the reduction of the degrees of freedom in which the human mind and body can play? Almost by definition, almost as the criterion for its existence, a society attempts to make itself less dependent on each of its individual components. So, in a society composed of humans, what would be a fairer mechanism for dissolving these snowflake dependencies than the invention or discovery of techniques by which to make the system as a whole less dependent on any possible human input?
Maybe "disempowerment" is a bit of a red herring, or a misleading problem to focus on. The reason we didn't spend more time on clarifying that is that we're just using it to gesture towards a different set of mechanisms that lead to extinction-like outcomes than usual. So even if you think our definition of empowerment is poor, or that empowerment isn't a great goal - that's kind of OK.
The thing we want to emphasize is that right now there are some mechanisms that steer our civilization towards keeping us alive, and free in some senses, that might stop operating. Though I take your point in the last paragraph that this might not change things much specifically regarding the implicitly empowering forces. We'll think about this one some more!
I agree with this formulation. What I am emphasizing is that, insofar as mechanisms are steering, the system in which they are operating can be said to have largely decoupled from the human mind.
"Alive and free in some senses."
A society whose tagline is "alive and free in some senses" is already dystopian! Far scarier, to me, than its extinction or my own early death.
Haha. Well we might agree about that - that description covers a wide range of possibilities. If you have ideas about what a plausible and good future looks like, please let me know. One of my next projects is trying to articulate "what is the best we can hope for?" and talk about which sets of goals are even possible to jointly fulfill. But certainly "everyone is free in all senses" is incoherent or at the very least, unstable.