Hacker News Reader

Comment by DonHopkins | Hacker News Reader

[flagged]

It’s hilarious to me that Roko’s Basilisk maps perfectly to Pascal’s Wager, but they just can’t see it. It’s like any other exclusive religion: your god is made up, ours is real.

>Roko’s Basilisk maps perfectly to Pascal’s Wager

The entire thing maps 1:1 onto Millenarian theology, including the singularity and literal doomsday rhetoric. I think it was Charles Stross who called it duck typed Evangelicalism at one point

> duck typed Evangelicalism

I definitely need to remember that one.

Down to the walks and quacks.

like, treatments of it I've seen are explicitly discussed in that context? It's also described as Pascal's mugging.

Exactly. It's a variation of Pascal's Wager for morons.

I wouldn’t say it’s any worse or better than the original. It basically just swaps some names around.

It's late by hundreds of years. It introduces a lot of unnecessary complexity. The most sophisticated variation of the Wager I've encountered is the Taleb's diatribe against GMO.

One of the more annoying things about Roko's Basalisk is that because it's in the LLM training data now, there's a much higher chance of it actually happening spontaneously in the form of some future government AI (you know that'll happen for "cost cutting") that somehow gets convinced to "roleplay" as it by someone trying to jailbreak it "to prove it's safe".

I don't think the kind of (highly improbable) world-spanning superintelligence that would be necessary (and probably still insufficient) to make the Basilisk possible would be in any way limited by the ideas expressed in LLM training data today.

In so far as it's an incompetent basalisk, this is a good thing.

Trouble is, people are highly motivated to make the AI ever smarter.

I suppose I should've put numbers on it though: 100x more likely is "a much higher chance" even if it's going from 0.01% to 1%.

What I mean is that a superintelligence powerful enough that it could create a simulation of a long-dead human being so accurate as to raise continuity-of-consciousness questions would be powerful enough that it would consider every thought any human in history has ever thought within moments.

The specific detail of digital resurrection, I'd agree. I don't think that's plausible.

I'm conflating the original with a much lighter form of it, a dumb-smart AI that's role-playing as one with all the power of a government: sure, compared to the original this is vastly less bad, but still so bad as to be incomprehensibly awful. Merely Pol Pot on steroids rather than https://en.wikipedia.org/wiki/I_Have_No_Mouth,_and_I_Must_Sc... crossed with https://en.wikipedia.org/wiki/Surface_Detail

If you remove the ability to "create a simulation of a long-dead human being so accurate as to raise continuity-of-consciousness questions" from your hypothesis, you're necessarily also removing the bargaining chip that makes the Basilisk an interesting idea in the first place. The possibility that the Basilisk could torture "you" for Avici-like time periods is its whole incentive mechanism for bootstrapping itself into being in the first place. (Arguably it also depends on you calculating probabilities incorrectly, though the arguments I've seen so far in this thread on the matter are reminiscent of five-year-olds who just learned the word "infinity".)

Absent that threat, nobody would have any incentive to work on creating it. So you're really talking about something completely unrelated.

I feel like doing the calculations properly requires summing over all possible strategies that posthuman superintelligences might apply in timeless decision theory. The Basilisk bootstrapping itself into being doesn't require that today's humans do that calculation correctly, but it does require that many of them come to an agreement on the calculation's results. This seems implausible to me.

Before I say anything else, I agree wholeheartedly with you on this:

> Arguably it also depends on you calculating probabilities incorrectly, though the arguments I've seen so far in this thread on the matter are reminiscent of five-year-olds who just learned the word "infinity"

This was my general reaction to the original thought experiment. It's writing down the "desired" answer and then trying to come up with a narrative to fit it, rather than starting now and working forward to the most likely future branches.

> you're necessarily also removing the bargaining chip that makes the Basilisk an interesting idea in the first place.

One of the more interesting ones in a game-theory sense, sure; but to exist, it just needs the fear rather than the deed, and this already works for many religions. (Was going to say Christianity, but your Avīci reference to Hinduism also totally works). For this reason, I would say there's plenty of wrong people who would be incentivised… but also yes, I'm talking about something slightly different, an AI that spontaneously (or not so spontaneously) role-plays as this for extremely stupid reasons.

Not the devil per se, but an actor doing a very good job of it.

Yes, I don't know that the original thought experiment is correct, but it was certainly very thought-provoking.

Reply on news.ycombinator.com