Can you find any explicitly EA organization that is at all aligned in perspective with anything you suggested?
Working to build AI at the company that is most publicly discussing the risks posed by AI to humanity and most advocating for guardrails (while getting an immense amount of flack for implementing them independently) doesn't seem to support anything about what you, or rather the other person said, about sacrificing humanity for anything AI related.
Anthropic are explicitly building ASI, using recursive self improvement, with no knowledge of how to control it. They spend only a minimal fraction of their funds on safety work, none of which relates to controlling ASI.
I am open to being wrong about Anthropic, but it will be hard to convince me that a company developing ASI is prioritising safety. Do you really believe what they are doing is safe?
It's kind of like saying you can't kill everyone with weapons when you're looking at what you can buy at the local Walmart and not what's in the 1000 foot deep missile silo.
Also, it's kind of a once you build it, it's too late kind of issue. No model will be good enough, the problem is still in the future, we're all dead.
Instead, he is turbocharging development, otherwise said choosing to cause harm he warns about.
That is literaly example of causing harm now and in near future, not caring about people now, only about vision.
He works on the safety team and their responsible scaling policies. He literally may be working on slowing down AGI progress at Anthropic.
Hiring own husband to safety team while you raise money for (their words!) doom machine, while you do your utmost to scare people while you try to raise even more money from investors on doom ... does not show OP was not correct in their assessment.
Just like the utilitarians in the thought experiment tossing everything positive in their life into the bottomless pit of suffering, the effective altruist AI thought leaders are tossing our entire economy to the gambit that the singularity will bring about the end of human suffering.
Capitalism has always been a tension between capitalists, who want maximum return on capital and workers, who want pesky things like a living wage, sick leave or safe work conditions.
For the longest time, the only power labor has had to get those things was the power of collective bargaining. No agreement with your workers meant no production happened.
Now, there's finally a chance at salvation. The capitalist Messiah is AGI and it will finally deliver them from those annoying laborers.
This ideological bent is why they're putting everything they have into AI. It's why VCs and their fellow capitalists are going so crazy.
They see it as a way to finally solve the contradictions of their ideology, but in reality it'd only create a new one: If everyone's out of a job, who will buy their products?
It doesn't matter that I can't do it. They'll pay me lots of money if they think I can.
AI will make human thought economically useless. In the new economy, the one economically useful thing we will be able to do is legal property ownership.
This is correct. But we don't need to ban AI to avoid the permanent underclass nightmare.
It's time for the means of production to be owned communally.
It's either a cybersocialist utopia or a neofeudalist dystopia.
Rosa Luxemburg formulated it pretty well more than a hundred years ago: Socialism or Barbarism!.
Today, people have leverage because removing them has real economic consequences. With AI, we need to be on the useful side of the divide to have any leverage.
And then we'd better hope AI has respect for property rights.
It's probably a lot easier to convince the military AIs we're building to do it.
I don't particularly like the situation, but I plan to do what I can to be on the winning side. The world is going to be what we're all in on building, and for people on the winning side it may be a utopia. Either way, I know which side of the gun I want to be standing behind.
Also note that an AI well aligned with the people who initially trained it wouldn't need to actually kill anyone that didn't take action against the owners, it would just need to outcompete people for resources, and let them fend for themselves in extreme poverty. There's no need to act one way or the other beyond that.
I'm talking about morality here, not practical ability. The ultrarich have their circle of friends and family, who themselves have their own social circle, on and on until suddenly you're not looking at wealthy people anymore. Six degrees of separation and all that. How many people in the social graph can know about this extermination plan before the whole thing collapses?
It's just disenfranchisement. Only the people who try to use violence to replace the political leverage they lose when they lose economic leverage would need any action taken against them, and that's easy to justify as self defense.
I think it's inevitable.
When AGI becomes reality, that reasoning disappears. When everyone's worthless, no one is. Thus the scenario you speculate about would need everyone in the upper echelons of society to be consciously, openly sadistic power-hungry people, which is not the case even when taking into account the higher dark triad rates among them.
Why did you think there's a massive uptick in surveillance and policing technology and automation? It's to make the French solution unfeasible.
What does this mean in practice? That an individual cannot own a machine (which is a means of production)?
As a first intuition, I feel like its mostly server farms and data centers that are the concern, large scale means of production.
It feels right that those would be communally owned, be that via democratic government or maybe some sort of worker owned coop?
Nothing prevents the government or coops today from building these things.
Permanent underclass is ideological goal of the above people. AI is an impersonal technology with no goals or wishes.
There's an inevitable effect that it has, and the only lever you have as an individual is being on the right side of that.
Even if the US federal government refuses to act, smaller government entities can deploy their own compute clusters running strong open-weights models like Kimi K3, and provide their own citizens with free access to strong AI models. There may come a time when even Chinese providers no longer release strong model weights, but then the previous generation of strong open-weights models can be used to accelerate the development of data pipelines and training setups to build the next generation. California or Australia (or many others) could fund university-led R&D efforts to build completely open models.
Despite the angry noises that leading providers make about their model outputs being used to train competing models, that's not illegal. States can use Common Crawl, Anna's Archive, Books3, commercial model outputs, and whatever other data sources that are useful for building models. States have sovereign immunity to ignore copyright complaints, if they decide to pursue model development.
The big AI companies don't have much of a moat. Let the big companies spend a fortune figuring out what works best, then openly build imitations having 95% of the capability at 5% of the cost.
AI is the least of the problem.
Can't help wondering if they've really thought this through.
That was the exact line of thought, coupled with resources are limited (marketed as climate change), that led to the 1 child policy in China and many other decisions that shall not be spoken of.
The underlying purpose of AI is to allow wealth to access skill while removing from the skilled the ability to access wealth. (read on HN, not my words)