Hacker News Reader

Comment by dvzk | Hacker News Reader

I admit I’m completely out of my depth when it comes to this field — I don’t even typically care about AI — but Eliezer’s response looks really bad to anyone in research. Asking for a citable and thorough written argument is as basic a requirement as it gets.

To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less!

I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.

View on HN

As someone who has outgrown that particular community, Yudkowsky has a talent for acting in bad faith argumentatively while not appearing to do so. He relies heavily on convservational implicature and passing along the burden of proof (as he did here) with no intention of being helpful to those that can generally improve his case through criticism or a distinctive aproach, as it would detract from sustaining his perceived position a focal or pioneering contributor. He should be applauded as a popularizer and point of inspiration, but his duplitousness in his argumentative style, once I identfied it, always left a bad taste in my mouth.

This seems like it's just an issue with the way conversations look on Twitter. If this were two people walking down the street and talking, I don't think that would be taken as a repudiation, just a request to clarify a vague question.

Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf

He doesn't want to open himself to criticism from people that might actually do a good job of it.

Did he tell you this? That seems like a mean-spirited accusation.

You would have to deal with the community's pseudointellectualism long enough to identify it as a pattern. Taken in isolation it looks like nothing, but the arc of the rationalist community has been insularity combined with CYA until a crisis comes along which makes them both more guarded with others while appearing to "update" in public discussion, using such a limited notion of belief because they view their core texts and blog posts as a canonical view of philosophy, and covering up degenerate behavior like sexual abuse on a regular basis. The real rationalists just move on from this crazy making. I've had to decondition myself like an ex-cultist which is why I'm reacting this way and I hope others can see how he behaves problematically too.

What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist? Obviously they are dangerous for the world because we might kill everyone if someone makes a mistake or gets mad and shoots one first. The danger of nukes is more obvious than why AI is dangerous.

> "What's the academically supported description/ argument or whatever for why nuclear weapons are dangerous for the world if they exist?"

The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".

Well THAT support starts with the experimental records of Manhattan Project and fills a whole library. Support for current ML techniques being dangerous in that sense are handwavy descriptions of what might well be science fiction for another 500 years.

Well, support for ML techniques being stupidly wrong is much bigger. It's the default state.

Then there's some less support for them being misleadingly wrong, much more rarely.

Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.

Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.

Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...

And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.

A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)

What about the Non Proliferation Treaty?

The nuclear non-proliferation treaty has somewhat limited nukes. But now Iran is on the verge, Israel, India, Pakistan , North Korea and other countries must be pursuing them. North Korea has shown the value of them because nobody can attack them or stop them because they do have some nukes and missiles. Israel and Iran are always attacking each other and regardless of the reason or goodness of either side, I'd pick them as the two most likely countries to start a nuclear war. Maybe even worse than Pakistan versus India.

I'd say we need a new strategy on top of the nuclear non-proliferation treaty.

What Chalmers is quoting is a tweet from Yudkowsky claiming that “ruin” is the guaranteed result of inventing AGI.

If someone said “the invention of nuclear weapons is guaranteed to lead to global destruction,” you might ask to see their reasoning for such a strong assertion. (As opposed to your assessment of nuclear weapons as a risk to be managed.)

Sorry, I don't understand. Are you saying that AI is of the same sort of danger as nukes?

"The danger of nukes is more obvious than why AI is dangerous."

Succinctly stated. This is why people are asking for canonical arguments for AI Ruin.

First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them. So I was raising the issue of nukes as obviously dangerous, ai less so, we can consider the dangers of AI or anything in the abstract.

I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.

There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.

> First, I was surprised about the idea that there must be an exact description of dangers before someone can investigate the dangers and consider them.

Of course there has to be a description of dangers before someone can consider them, otherwise how do we evaluate risk? How else do we know whether we are overreacting? It's not like the "ruin" people suddenly woke up yesterday morning in a panic. They have been talking about this for years. In all that time, there has been no dispassionate but thorough analysis of the dangers?

This is what social media’s real damage is. It turned academic progress into arguable points. Your research degree and countless lab hours is equally valid to my two-second no-research hot take, and if you’re not on board with it then you’re swarmed.

> Your research degree and countless lab hours is equally valid to my two-second no-research hot take,

But you don't understand, I did my own research

> my two-second no-research hot take

This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it.

Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky

And Chalmers is a philosopher focusing on consciousness and cognitive science – https://en.wikipedia.org/wiki/David_Chalmers

I wouldn’t put their replies in exactly the same category as your typical social media reply guys.

I didn’t mean my comment about this tweet exchange as an example, I was only trying to speak to the comment I replied to about a general trend. This particular exchange is not the problem; it’s the inability for the public to gauge when it’s not like this, though, if that makes sense.

Yudkowsky and LessWrong was regarded as kooky for ages and, after the Roko's Basilisk affair (https://en.wikipedia.org/wiki/Roko%27s_basilisk and read Yudkowsky's hilariously deranged all-caps response), a laughingstock as well. Suggesting that he's on the same plane as serious researchers is inappropriate.

> This is what social media’s real damage is. It turned academic progress into arguable points.

If that is "damage", the problem must surely lie with academia, not with social media.

Academia's only raison d'être is to facilitate formal discourse. Wherever it doesn't do that, it loses its legitimacy. "I'm a researcher and you are not" is not an argument. In fact, I'd go so far as to claim that anyone who makes such an "argument" should be excluded from any future discourse.

I think you may be misunderstanding the GP (though it's possible I am); my reading of that was not that it actually changed what "academic progress" is, but rather that it changed the public perception of it, as it gets discussed publicly on social media, where (particularly on Twitter) it's much more important who can draw the most views/shares/etc than who actually has the better science.

This field has created a lot of writings but, as far as I can tell, no knowledge. The subject matter of this field does not exist. I don't think you need to know much of anything to match the supposed experts. Watching Terminator is probably good enough.

Reply on news.ycombinator.com