Anthropic is a company full of basilisk believers.
a) believe that what they're creating is a basilisk, and b) keep trying harder to do this while staring right at it
I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)?
That seems to be why this individual resigned, but I'm surprised it's not all of them. The cakeism is strong in that company.
In short, this is the believe that a god-like AI could punish them retroactively, for not having done all that was in their power to create this AI.
(A bit similar to some religious believe that a god could punish you after your death if you did not spend your live "pleasing" said god during your life)
(I am not a basilisk believer. I think this is all absolute horseshit. But to understand someone's motivations, one must think like them.)
Only if you don't spend 5 minutes thinking about the Self as a concept. Resurrecting someone by perfect simulation is a cool plot device for a novel like Altered Carbon (which ironically is really about wealth inequality) but not a serious hypothetical.
Even if we ignore that, no, super intelligent MechaGrok17 cannot make a perfect copy of you posthumously from your embarrassing Character.AI chats and IoT toilet data, the copy it still wouldn't be YOU. It would (supposedly) think and behave like you, but for all purposes it would be the Basilisk torturing a future stranger, or your descendants. I'm very much a fan of people not being potentiality tortured, so the rational action is to stop the creation Basilisk, or even just delay it so you aren't around for it. There's a bunch of other big assumptions that makes Roko's Basilisk really stupid, but they are unimportant. It's just theory crafting, like talking about the lore of video games or marvel comics. It doesn't help you understand their real motivations.
Roko's Basilisk is just recontextualized protestant Hell in a sci-fi coat of paint. It's the childhood anxieties of American nerds about hell and sin. The problem is that they are still culturally protestant. God doesn't exist or he's hands off, but in their minds there's still a open position for the wrathful and proactive God. If you think that you actually can create God for realsies, your very rational™ prediction of its behavior will coincidentally line up with your childhood fears.
Why is the spooky AI mad at you? Because you didn't worship it correctly.
Why do you obey it? Because if I don't it will torture me for eternity.
How will it do that? God is omnipotent, don't think about it.
Why don't try to stop it's creation? “I am the Alpha and the Omega,” says the Lord God, “who is, and who was, and who is to come, the Almighty.”
All the scifi scaffolding and stuff like acausal trade is paint over Timmy's first panic attack on Sunday school 30 years ago.
However, the version that says AGI—as in, the Basilisk—will be created within our lifetimes proposes that it will punish the actual person.
All that said, I will reiterate that I think the whole thing is utter nonsense.