Hacker News Reader

Comment by Vecr | Hacker News Reader

It's probably the best approach known so far, but for all their talk of "long termism", I predict many people will dislike the fact that they'll almost certainly be dead (and in an unrecoverable state, before anyone gets their hopes up) before the project gets completed after the 40+ years and trillions of dollars it will take even in the optimistic case, i.e. for them personally there's no difference between no AGI/ASI and aligned AGI/ASI, as it will take too long.

Arguments about supposed "races with China" will probably be deployed against this idea, but the only thing the leaders of the AI labs will be racing against is their own deaths and each other.

View on HN

> It's probably the best approach known so far

Why do you think that?

I love this line from the paper:

> "So, if a person or organization wants to be sure that their AGI never lies, never escapes and never invents bioweapons, they need to impose those requirements and never run versions that don’t provably obey them."

Oh is that all.

It's the best option, even assuming no one raced and you got full funding and enthusiasm, it would be plausible to put success at 40%. That's after a generation of work and trillions of (2022, probably) dollars, but it's sure better than the ~0% chance I'd give anything else. The coordination problem for getting AI using this method and the coordination problem to prevent anyone getting AI at all is quite similar though, but probably somewhat more preferable.

> It's the best option

That's not an answer.

> it would be plausible to put success at 40%.

No, it wouldn't. You just made up that number based on nothing.

We have a good amount of experience with provability in software, and its limitations are well-known. Tegmark et al. seem to me just to be waving the magic pixie dust of "AI will solve this", and making unlikely claims, with no real substance.

I was asking what about what they're saying makes you think that "it's the best option", but it's clear I'm not going to get a useful answer from you.

If it's not the best option, what is? It's the only method where I think you would be able to detect failures before the AI system is running. RLHF and "Constitutional AI" absolutely can't do that with current neural network analysis methods.

> I predict many people will dislike the fact that they'll almost certainly be dead (and in an unrecoverable state, before anyone gets their hopes up)

Cryonics is a thing already.

Yeah, I don't know. Gwern thinks it would work if done near perfectly, but in practice what gets done is freezing dead bodies in an already pretty much unrecoverable state (if not already dead and warm for hours if not days), with no precautions (or only "feel good" non-functioning precautions) against microscopic ice spike formation, and then at some point someone screws something up and you get a temperature excursion and then there's nothing left information recovery wise. I don't agree with Gwern that current techniques would work even in theory.

Cryonics is a backup mechanism where nobody has yet implemented the ‘restore’ function.

There are meat popsicles in freezers, but they'll almost certainly never be conscious again.

>for them personally there's no difference between no AGI/ASI and aligned AGI/ASI

Not really true. There are things worse than death and every AI researcher who is width their salt knows this.

Human level intelligence already inflicts not only great amounts of pain but drawn out, unusual, and unnatural durations of pain. So we know artificially enhanced suffering is a risk of intelligence.

I did not say there was no difference between the other pairs, obviously there's a difference between aligned AI and no AI, and between unaligned AI and aligned AI, I'm saying if aligned AI takes so long to create that you'd be dead first, to you personally it's the same as it not existing.

Reply on news.ycombinator.com