If they believe it, they can stop. All of them. There's no sign though that any pacing/slowdown or whatever is in place, zero.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
As for the other point, crazy, cynical, and even unethical as it may come off, it is hardly inconsistent with how people generally behave when met with hard limits (i.e. making the most of what's available, and approximating the desired state if that is unattainable). This is just a variation on the same.
Something could be said about "what if they're mistaken", but if all players are making a mistake, then that makes little practical difference; it may as well be real.
Hence the argument that they're either full of shit, or psychopaths who need to be stopped at all costs.
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
They want coordinated action to prevent a multipolar trap and potential catastrophe by future superintelligent AI (which may emerge quite rapidly if RSI ever hits an inflection point). It's not rocket science. The rules would apply to everyone. The rules they're advocating for are not "only OpenAI and Anthropic can make AI".
Critics of Yudkowsky have been trying to misportray that for years to make him seem delusional. You can see an example of that here: https://www.tracingwoodgrains.com/p/reliable-sources-how-wik...
(Roko himself is genuinely crazy, but largely because he became a neo-Nazi.)
If you think ASI is plausible this century, you would have no reason to not fear the end times yourself. If you don't think it's plausible it'll be created this century or perhaps ever, fair enough, but I'd bet against that.
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Ref w/gift link https://x.com/sapinker/status/2096236079477096630
Most people among both ideologies have long believed in the arrival of a machine demigod of sorts, but I kind of am skeptical of the notion that there are any tech people who have not always held similar beliefs in the long run. Like, all the people reading this: you think in a thousand years from now humans are going to be doing most labor? Obviously far, far, far smarter machines will have unimaginably changed things by then, and/or we'll all be gone.
The difference is most people in tech have historically thought it's maybe a century or three away, while the rationalists think it's a decade or two away (and also thought it would be 2020s - 2040s even decades prior to now).
Ladies and gentlemen, I give you your totally-not-in-an-insular cult members.
A thousand years is a long time, and the world will be so different that it's hard to make any concrete predictions. But yes, if I had to wager, I'd bet that what humans conceive of as "labor" in 3026 will still be whatever difficult or annoying bits are done by humans.
In 1026 nearly all labor was agricultural labor. If you told a Roman peasant that magical self-driving contraptions could sow and reap the fields, he might wonder what could possibly be left for humans to do in such a world. And yet, here we are, and I still have to update JIRA tickets.
For my myself, I expect AI to prove a more transformative technology than Docker but less transformative than the sewing machine.
What you think won't happen in 1000 years is what I think is almost certainly happening within 40 years. I'd bet everything I own it'll be way, way, way more impactful than the sewing machine within 40 years. I think some bits of physical labor may still exist in 40 years, but almost none, and only when a human really wants to talk to or deal with a human, such as in some medical and therapeutic professions, some art fields, etc. (Again, assuming no universal outlawing of AI.)
Why do you think restaurants with waiters keep opening when the automat has existed since 1895? https://en.wikipedia.org/wiki/Quisisana
You are also probably unfamiliar with the history of the sewing machine and its impact on society. As AI has already poisoned search engines, here's an AI slop overview that hits some of the main points: https://sciencepedia.blog/sewing-machine-societal-importance
Yes, I think even in some crazy distant future utopia, humans will still often want to interact with humans for things, and even to pay them for some services. Tasks that include speaking one-on-one and producing works of art will always retain some value to people, even when AIs can one day trivially produce things which most humans find aesthetically superior to anything a human could ever produce alone. People will seek other things out of art beyond aesthetics, like expressions of human emotion.
Humans will enjoy and seek out human contact. This is not surprising. Just, for economy-impacting labor, I think it's almost all going to be machines, with very few humans playing any role. Even an oversight or accountability role. I think there's a 95% chance of it by 2100 and an 85% chance of it by 2066, conditional on no laws that forbid replacing jobs with AI.
You're making the "this time it's different" case that AI will be categorically different from every other form of labor-saving automation that's come before it. I work with these models every day and I'm not buying it.
(I personally lean towards "Yudkowsky is basically correct about everything, but we should probably try to build ASI anyway, slowly and carefully". I'm pretty much where Amodei's at. I assume Altman has a similar idea, though is a bit less pessimistic.)