Hacker News Reader

Comment by mcherm | Hacker News Reader

This has been on HackerNews before, but it is still interesting. It is also worth noting that in http://lesswrong.com/lw/up/shut_up_and_do_the_impossible/ he admits he has conducted 3 more experiments since then (for more money) and was successful in one of those. The fact that it was EVER successful (using a mere human, not a smarter-than-human AI) makes the point.

View on HN

It's not really much of an experiment if you refuse to publish your methods and data. That's pretty much the opposite of science.

He did publish his methods (how it was set up, what the rules were, etc) and data (they let him out on X tries), just not the data that would interfere with the ability to do the experiment again (e.g. his exact strategy).

Not much different, in principle, from not publishing the names of people who participated in drug trials.

No, it's very different from that. It's more along the lines of demonstrating a drug that cures cancer, but refusing to tell anyone its chemical composition or how to make it.

If the purpose of your research was only to establish that there's a (nontrivial) "risk" of someone curing cancer (as Yudkowsky was trying to establish that there's a risk of an AI talking itself out of a sandbox), then yes, that would be sufficient, assuming the patients actually went into remission with higher than usual frequency after your interventions (as Yudkowsky's subjects unboxed the AI with higher than usual frequency).

But he could be cheating. He could literally be telling these people "I'll give you a thousand dollars if you let me out and keep the conversation a secret."

It could even be worse than that. The people could just be his friends, or alt accounts (unlikely).

I have heard about this several times and I find it extremely difficult to believe that this is real. Not that I doubt that a superhuman AI could possibly convince people to let it out, but I don't believe that a human, no matter how persuasive, could convince another human over IRC to go against something that they have decided in advance when you know they are purposefully just trying to convince you of something that you don't believe.

The fact that none of the chat logs are released makes me only more incredulous. I would understand if the author wanted to do two or three trials with the same strategy which could be in some way ruined by revealing it ahead of time (which already seems implausible) but at this point there is literally no conceivable reason to keep this a secret other than that it is a sham.

...or that the chat logs being kept secret indefinitely was an important part of the strategy. After all, if the AI exploits some embarassing secret of yours to be let out, that wouldn't work if you knew the logs could be publicized some day. I think over-eagerness to claim things like "literally no conceivable reason" is one of the things that lets oddities like the box experiment work.

On the contrary, the guardians had to pay some amount if they let the AI out. The stakes varied greatly, but if I recall correctly Eliezer cashed in 3000 dollars (or something in that ballpark) from one guy.

Ok, then "I'll secretly give you $1000 more than what you're publicly giving me." Happy?

If it's a science "experiment", his strategy would have to be revealed so you can reproduce it. Names of people participating in drug trials is not required to reproduce an experiment. In principal this makes it different from not publishing the names of people who participated in drug trials.

All he has "proven" is that a certain subset of people can be conned into typing something into at terminal. I don't get the significance. For all I know, he's choosing his target.

The targets are, indeed, selected, by the criterion "You believe not even a transhuman AI could get you to let it out of the box."

I might believe a transhuman AI could convince me; I am not convinced that any human can emulate a transhuman AI well enough to do so.

Would you say that your winning strategies involved thinking transhumanly (perhaps in non-realtime, a la Vinge's Mailman)?

Obviously no, essentially by definition.

This is exactly what's been burning me about this whole "experiment" since I read about it a few months ago, and went looking (unsuccessfully) for what actually happened in the "tests". I find it hard to see this as anything more than some sort of self-aggrandizing publicity stunt.

Amazing experiment. I wonder if it cost Yudkowsky money to get out of the box. I think that a bribe is the only way he could convince me to let him “win” the contest.

Real-world bribes were forbidden in the rules, I think, so Yudkowsky couldn't have done that.

Reply on news.ycombinator.com