It is part of the fear - human intelligence is extremely limited. That could easily be representative (although to be fair, probably not).
Reasoning about the future is really hard and usually everyone who makes predictions is wrong. So far we've been fine through pretty much every challenge. That being said, "we'll survive this" is a statement that is only false once and this one looks pretty serious. We might be dealing with a world where human intelligence is economically uncompetitive for the first time in recorded history.
I find it tricky to think about cases like Yudkowsky (full disclosure I used to read LessWrong a lot), because if he has sufficient credibility then loudly staking your extreme position can indeed move the Overton window.
If Yudkowsky is moving the Overton window, it's in a direction opposite from what he intends.
"Coherent Extrapolated Volition (CEV) is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted."
We should, as a society, offer help to people who are caught in the cynicism trap, because we could still have all those things Eliezer sees as our better selves. We just have to spend more time focusing on being on our better selves and less time giving up on each other.
Deleted comment
Even if you still have the cynical perspective of thinking it's all wrong--Christianity took over and nobody calls it a cult anymore. At least give it the dignity of 'religion'.
This is the most punk rock idea I've heard in awhile.
Yudkowsky is a prime example of the latter group. I think I became aware of him through the Roko's Basilisk meme many years ago, and I thought "wow so this is what happens when you have a lot to think about and nothing to do". It's basically nonsense.
Two other people like that are Nick Bostrom and Will MacAskill.
I heard of "Superintelligence" by Bostrom probably through an Elon Musk recommendation in 2015 or so. That's when he started OpenAI, and I was working on AI at the time. (Remember Elon had a completely different reputation then; he was respected as a person who built things, and wasn't associated with any political ideas.)
So I got the book, and to its credit, the beginning of the book acknowledges that the rest of the book may be complete bullshit. It's extremely low confidence extrapolation. And I pretty much agreed with that preface -- most of the book was nonsense. However many people seemed to take it seriously.
(I'm not saying AI is going to be great, or catastrophic either. I was just looking for some kind of high quality analysis from someone with domain knowledge, and found none of it in that book.)
----
I think I became aware of MacAskill through a NYTimes article puffing his book last year. There was something "off" about it.
And few months later, it turned out that he was conveniently unaware that the only reason that his projects were funded was because his friend was committing a huge crime:
https://time.com/6262810/sam-bankman-fried-effective-altruis...
So yeah, obviously I can't take seriously the moral opinions of somebody whose morals fall down immediately and spectacularly when faced with a whiff of the real world.
It seems like his main job is to create and advocate ideas that are intellectually appealing to and in the interest of billionaires. It came out in the FTX aftermath that he was telling Elon that SBF could help him buy Twitter.
BTW Elon also recommended the MacAskill book, which another thing that has lowered my opinion of Elon. People are people are flattering him with ideas, and he's taking the bait.
----
Anyway, normally I wouldn't have any awareness of the kind of writing that these 3 people produce. Much of it is low quality, which I would expect given their lack of experience and expertise in the domain of AI. And 75% of it couldn't have any effect on the world -- even in principle.
But like you, I'm puzzled that some people I respect seem to take them seriously. I mean I do respect Fridman, since I've learned a bunch of things from his podcasts with other guests. (I understand why a lot of people don't like him, but I prefer to focus on a person's best work and not their worst.)
But yeah the best work of Yudovsky, Bostrom, and MacAskill doesn't seem noteworthy.
It seems designed to generate attention, and that's mostly it.
The extremely, extremely obvious thing they're missing (or just fail to emphasize because it won't get them attention) is that you should be scared of corporations and governments with AI (first), not scared of autonomous AI with its own will.
That is, the "superintelligence" and "longterm-ism" ideas are basically deflections from the real issues with AI.
----
So yeah I was puzzled about this whole thing, and someone on HN recommended this substack, and I recommend it too. It's pretty interesting and it draws a line between the 90's extropians@ mailing list, Yudowsky, Bostrom, and multiple founders of Bitcoin.
Also a lot of dark stuff like suicides, mental illness, and abuse in the rationalist movement.
https://aiascendant.substack.com/p/extropias-children-chapte...
What also seems to have happened in this era is that Yudkowsky and MIRI, flush with newly donated millions, decided to try to evolve from theory and community-building to actual practical AI research. It did not go well.
This was quite a claim: essentially, that this research would, if published, meaningfully accelerate perhaps the most extraordinarily difficult challenge in the history of human innovation. It is a claim which remains 100% unsubstantiated.