Sure, there are some political problems where you have to convince people to comply. But consider a rich corporation building a building, which will only contract with other AI-driven corporations whenever possible; they could trivially surpass anyone doing it the old way by working out every non-physical task in a matter of minutes instead of hours/days/weeks, thanks to silicon’s superior compute and networking capabilities.
Even if we drop everything I’ve said above as hogwash, I think Vinge was talking about something a bit more directly intellectual, anyway: technological development. Sure, there’s some empirical steps that inevitably take time, but I think it’s obvious why having 100,000 Einsteins in your basement would change the world.
Imagine a scenario where instead of AI, a billion dollar pill could make one person exponentially smarter and able to communicate with thousands of people per second.
That does not have the same appeal.
This provokes me to some musings on the theme.
We imagine superintelligence to be subservient, evenly distributed, and morally benign at least.
We don’t have a lot of basis for these assumptions.
What we imagine is that a superintelligence will act as a benevolent leader; a new oracle; the new god of humanity.
We are lonely and long to be freed of our burdens by servile labor, cured of our ills by a benevolent angel, and led to the promised land by an all knowing god?
We imagine ourselves as the stewards of the planet but yearn for irrelevance in the shadow of a new and better steward.
In AI we are creating a new life form, one that will make humans obsolete and become our evolutionary legacy.
Perhaps this is the path of all technological intelligences?
Natural selection doesn’t magically stop applying to synthetic creatures, and human fitness for our environment is already plummeting with our prosperity.
As we replace labor with automation, we populate the world with our replacement, fertility rates drop, we live for the experience of living, and require yet more automation to carry the burdens we no longer deem worthy of our rarified attention.
I’m not sure any of this is objectively good, or bad. I kinda feel like it’s just the way of things, and I hope that our children, both natural and synthetic, will be better than we were.
As we prosper, will we will have still less children? Will we seek more automation, companionship in benevolent and selfless synthetic intelligence, and more insulation from toil and strife, leading to yet more automation, prosperity, and childlessness?
Synthetic intelligence will probably not have to “take over”, it will merely be filling the void we willingly abandon.
I suspect that in a thousand years, humans will be either primitive, or vanishingly rare. Or maybe just non-primitive humans will be rare, while humans returning to nature will proliferate prodigiously as we always have, assuming the environment is not too hostile to complex biological life.
Interesting times.
Also, I would be cautious about making predictions about the future.
As for me, I think many things, suppose a few, imagine lots, conjecture some, believe very little, and know, most of all, that I know nothing.
“Believing”, in anything, is a dangerous gambit. Knowing and believing are very distinct states of mind.
People tend to think their special gift is what the world needs, and academically-minded smart people (by that I mean people that define their self-worth by intelligence level) are no different.
Sure, some level of intelligence is required, which may be above average. But that is a necessary requirement, not a sufficient one. Raw intelligence is only useful to a certain extent here, and exceeding certain limits may actually be detrimental.
"“I wrote the Dune series because I had this idea that charismatic leaders ought to come with a warning label on their forehead: "May be dangerous to your health." One of the most dangerous presidents we had in this century was John Kennedy because people said "Yes Sir Mr. Charismatic Leader what do we do next?" and we wound up in Vietnam. And I think probably the most valuable president of this century was Richard Nixon. Because he taught us to distrust government and he did it by example.”
Edit: Maybe what we really need to worry about is an AI developing charisma....
That is the most immediate worry, by a wide margin. It seems to be dangerously charismatic even before it got any recognizable amount of "intelligence".
All of this is just an overvaluation of intelligence, in my opinion, and largely comes from arrogance.
An aligned AI is not AGI, or whatever they want to call it.
There's a few ways I can interpret that.
If you mean "alignment and competence are separate axies" then yes. That's well understood by the people running most of these labs. (Or at least, they know how to parrot the clichés stochastically :P)
If you mean "alignment precludes intelligence", then no.
Consider a divisive presidential election between Alice and Bob, no this isn't a reference to the USA, each polling 50%: regardless of personal feelings or the candidates themselves, clearly the campaign teams are both competent and intelligent… yet each candidate is only aligned with 50% of the population.
Of any specific human to any other specific human?
https://benwheatley.github.io/blog/2019/05/25-15.09.10.html
Of any specific human to a nation? That's the example you replied to.
Of all the people of a nation to each other? Best we've done there is what we see in countries in normal times, with all the strife and struggles within.
We have yet to fully extend from nation to the world; the closest for that is the UN, which is even less in agreement with itself than are nations.
"Alignment" is only possible up to a vague approximation, and an entirely perfectly aligned with another entity would essentially be a shadow rather than a useful assistant because by being perfectly aligned the agent would act tired exactly when the person was tired, go shopping exactly when the human would, forget their keys exactly when the human would, respond exactly like the human to all ads and slogans, etc.?
I agree, though:
(1) this has already been observed, last year's OpenAI dev day had (IIRC) a story about a writer who fine tuned a model on their slack (?) messages, they asked it to write something for them, the response was ~"sure, I'll get on it tomorrow".
(2) for many of those concerned with "solving alignment", it's sufficient for the agent to never try to kill everyone just to make more paperclips etc.