How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
Poverty is great for OpenAI
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
- all humans should, of course, have a say
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.
Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.
Also, banning superintelligence won't stop scientific progress in general.
Speak for yourself. A future like in The Culture novels sounds great to me!
In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.
I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.
Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?
If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.
Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?
However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.