Take, for example, 80,000 Hours, among the more prominent EA organizations. Their top donors (https://80000hours.org/about/donors/) include:
- SBF and Alameda Research (you probably knew this),
- the Berkeley Existential Risk Initiative, founded (https://www.existence.org/team) by the same guy who founded CFAR (the Center for Applied Rationality, a major rationalist organization)
- the "EA infrastructure fund", whose own team page (https://funds.effectivealtruism.org/team) contains the "project lead for LessWrong.com, where he tries to build infrastructure for making intellectual progress on global catastrophic risks"
- the "long-term future fund", largely AI x-risk focused
and so on.
Check your notifications
Doom scroll
Refresh
Refre
Ref
(Of course, the actual thing is more closely "Zizian style cultish insanity", which honestly has very very little to do with LessWrong style rationality either.)
Eliezer's career has always been centered around AI. At first Eliezer was wholly optimistic about AI progress. In fact, in the 1990s, I would say that Eliezer was the loudest voice advocating for the development of AI technology that would greatly exceed human cognitive capabilities. "Intentionally causing a technological singularity," was the way he phrased it in the 1990s IIRC. (Later "singularity" would be replaced by "intelligence explosion".)
From 2001 to 2004 he started to believe that AI has a strong tendency to become very dangerous once it starts exceeding the human level of cognitive capabilities. Still, he hoped that before AI starts exceeding human capabilities, he and his organization could develop a methodology to keep it safe. As part of that effort, he coined the term "alignment". The meaning of the term has broadened drastically: when Eliezer coined it, he meant the creation of an AI that stays aligned with human values and human preferences even as its capabilities greatly exceed human capabilities. In contrast, these days, when you see the phrase "aligned AI", it is usually being applied to an AI system that is not a threat to people only because it's not cognitively capable enough to dis-empower human civilization.
By the end of 2015, Eliezer had lost most of the hope he initially had for the alignment project in part because of conversations he had with Elon Musk and Sam Altman at an AGI conference in Puerto Rico followed by Elon and Sam's actions later that year, which actions included the founding of OpenAI. Eliezer still considers the alignment problem solvable in principle if a sufficiently-smart and sufficiently-careful team attacks it, but considers it extremely unlikely any team will manage a solution before the AI labs cause human extinction.
In April 2022 he went public with his despair and announced that his organization (MIRI) will cease technical work on the alignment project and will focus on lobbying the governments of the world to ban AI (or at least the deep-learning paradigm, which he considers too hard to align) before it is too late.
The rationalist movement began in November 2006 when Eliezer began posting daily about human rationality on overcomingbias.com. (The community moved to lesswrong.com in 2009, at which time overcomingbias.com became the personal blog of Robin Hanson.) The rationalist movement was always seen by Eliezer as secondary to the AI-alignment enterprise. Specifically, Eliezer hoped that by explaining to people how to become more rational, he could increase the number of people who are capable of realizing that AI research was a potent threat to human survival.
To help advance this secondary project, the Center for Applied Rationality (CFAR) was founded as a non-profit in 2012. Eliezer is neither an employee nor a member of the board of this CFAR. He is employed by and on the board of the non-profit Machine Intelligence Research Institute (MIRI) which was founded in 2000 as the Singularity Institute for Artificial Intelligence.
I stress that believing that AI research is dangerous has never been a requirement for posting on lesswrong.com or for participating in workshops run by CFAR.
Effective altruism (EA) has separate roots, but the two communities have become close over the years, and EA organizations have donated millions to MIRI.
He has no formal education. He hasn't produced anything in the actual AI field, ever, except his very general thoughts (first that it would come, then about alignment and doomsday scenarios).
He isn't an AI researcher except he created an institution that says he is one, kind of as if I created a club and declared myself president of that club.
He has no credentials (that aren't made up), isn't acknowledged by real AI researchers or scientists, and shows no accomplishments in the field.
His actual verifiable accomplishments seem to be having written fan fiction about Harry Potter that was well received online, and also some (dodgy) explanations of Bayes, a topic that he is bizarrely obsessed with. Apparently learning Bayes in a statistics class, where normal people learn it, isn't enough -- he had to make something mystical out of it.
Why does anyone care what EY has to say? He's just an internet celebrity for nerds.
Also, this page lists 3 ex-Googlers as being currently employed by Eliezer's org: https://intelligence.org/team/
Nisan Steinnon who worked for Google also did some research work for the Eliezer's org.
At best -- if one is feeling generous -- you could say he is a "philosopher of AI"... and not a very good one, but that's just my opinion.
Eliezer looks to me like a scifi fan who theorizes a lot, instead of a scientist. So why do (some) people pay any credence to his opinions on AI? He's not a subject matter expert!
Your comment is like dismissing someone who is opposed to human cloning on the grounds that he hasn't published any papers that advance the enterprise of human cloning and hasn't worked in a cloning lab.
That's not the full claim you were responding to.
You were responding to me, and I was arguing that Yudkowsky has no academic credentials, but also no background in the field he claims to be an expert on, he self-publishes and is not peer-reviewed by mainstream AI researchers or the scientific community, and he has no practical AI achievements either.
So it's not just lack of academic credentials, there's also no achievements in the field he claims to research. Both facts together present a damning picture of Yudkowsky.
To be honest he seems like a scifi author who took himself too seriously. He writes scifi, he's not a scientist.
https://www.youtube.com/watch?v=xjH2B_sE_RQ
His book _Rationality: A-Z_ is widely admired including by people you would concede are machine-learning researchers: https://www.lesswrong.com/rationality
Anyway, this thread began as an answer to a question about the community of tens of thousands of people that has no better name than "the rationalists". I didn't want to get in a long conversation about Eliezer though I'm willing to continue to converse about the rationalists or on the proposition that AI is a potent extinction risk, which proposition is taken seriously by many people besides just Eliezer.
He was writing fan fiction and creepy torture shit one for ages until there was big money in influencing public policy on AI.
>He’s basically a PR person for OpenAI and Anthropic
How in the world did you arrive at that belief? If it was up to him, OpenAI and Anthropic would be shut down tomorrow and their assets returned to shareholders.
Since 2004 or so, he has been of the view that most research in AI is dangerous and counterproductive and he has not been shy about saying so at length in public, e.g., getting a piece published in Time Magazine a few years ago opining that the US government should shut down all AI labs and start pressuring China and other countries to shut down the labs there.
He is a "research fellow" in an institution he created, MIRI, outside the actual AI research community (or any scientific community, for that matter). This is like creating a club and calling yourself the president. I mean, as an accomplishment it's very suspect.
As for his publications, most are self-published and very "soft" (on alignment, ethics of AI, etc). What are his bona fide AI works? What makes him a "researcher", what did he actually research, how/when was it reviewed by peers (non-MIRI adjacent peers) and how is it different to just publishing blog posts on the internet?
On what does he base his AI doomsday predictions? Which models, which assumptions? What makes him different to any scifi geek who's read and watched scifi fiction about apocalyptic scenarios?
I've seen the explanation that AI pessimism helped Eliezer attract donations, but that does not work because his biggest donor when he started going public with his pessimism (2003 through 2006) was Peter Theil, who responded to his turn to pessimism by refusing to continue to donate (except for donations earmarked for studying the societal effects of AI, which is not the object of Eliezer's pessimism and not something Eliezer particularly wanted to study).
I suspect that most of the accusations to the effect that MIRI or Less Wrong is a cult are lazy ad-hominems by people who have a personal interest in the AI industry or an ideological attachment to technological progress.
AI alarmism itself isn't a well founded argument.
I don't think that's entirely true. A well-founded argument against AI alarmism is that, from a cosmic perspective, human survival is not inherently more important than the emergence of AGI. AI alarmism is fundamentally a humanistic position: it frames AGI as a potential existential threat and calls for defensive measures. While that perspective is valid, it's also self-centered. Some might argue that AGI could be a natural or even beneficial step for intelligence beyond humanity. To be clear, I’m not saying one shouldn’t be humanistic, but in the context of a rationalist discussion, it's worth recognizing that AI alarmism is rooted in self-preservation rather than an absolute, objective necessity. I know this starts to sound like sci-fi, but it's a perspective worth considering.
How very zizian of you.
A combination of a psychological break when his sibling died and that being a doomsayer brought him a lot more more money, power, and worship per unit of effort and particularly per unit of meaningful work-like effort.
It's a lot easier to be a doomsayer bullshiter than other kinds of bullshitters, the fomer just screams stop the latter is expected to accomplish something now and again.
I thought someone would bring that up, so I attempted to head it off in the second paragraph of this comment: https://news.ycombinator.com/item?id=42904625
He was already getting enough donations and attention from being an AI booster, enough to pay himself and pay a research team, so why would he suddenly start spouting AI doom before he had any way of knowing that doomsaying would also bring in donations? (There were no AI doomsayers that Eliezer could learn that from when Eliezer started his AI doomsaying: Bill Joy wrote an article in 2000, but never followed it up by asking for donations.)
Actually, my guess is that doomsaying never did bring in as much as AI boosterism: his org is still living off of donations made many years ago by crypto investors and crypto founders, who don't strike me as the doom-fearing type: I suspect they had fond memories of him from his optimistic AI-boosterism days and just didn't read his most recent writings before they donated.
He spoke to businessmen posing as experts, became increasingly self-referential, and frankly the quasi-religious subtext became text.
Here is an example (written in year 2000) of Eliezer's hyping of AI:
>The Singularity holds out the possibility of winning the Grand Prize, the true Utopia, the best-of-all-possible-worlds - not just freedom from pain and stress or a sterile round of endless physical pleasures, but the prospect of endless growth for every human being - growth in mind, in intelligence, in strength of personality; life without bound, without end; experiencing everything we've dreamed of experiencing, becoming everything we've ever dreamed of being; not for a billion years, or ten-to-the-billionth years, but forever... or perhaps embarking together on some still greater adventure of which we cannot even conceive. That's the Apotheosis. If any utopia, any destiny, any happy ending is possible for the human species, it lies in the Singularity. There is no evil I have to accept because "there's nothing I can do about it". There is no abused child, no oppressed peasant, no starving beggar, no crack-addicted infant, no cancer patient, literally no one that I cannot look squarely in the eye. I'm working to save everybody, heal the planet, solve all the problems of the world.
http://web.archive.org/web/20010204095500/http://sysopmind.c...
Another example (written in 2001):
>The Plan to Singularity ("PtS" for short) is an attempt to describe the technologies and efforts needed to move from the current (2000) state of the world to the Singularity; that is, the technological creation of a smarter-than-human intelligence. The method assumed by this document is a seed AI, or self-improving Artificial Intelligence, which will successfully enhance itself to the level where it can decide what to do next.
>PtS is an interventionist timeline; that is, I am not projecting the course of the future, but describing how to change it. I believe the target date for the completion of the project should be set at 2010, with 2005 being preferable; again, this is not the most likely date, but is the probable deadline for beating other, more destructive technologies into play. (It is equally possible that progress in AI and nanotech will run at a more relaxed rate, rather than developing in "Internet time". We can't count on finishing by 2005. We also can't count on delaying until 2020.)
http://web.archive.org/web/20010213215810/http://sysopmind.c...
No longer is he hyping AI though: he's trying to get it shut down till (many decades from now) we become wise enough to handle it without killing ourselves.
It would be less disingenuous if you were to say EA is the "it's objectively better for humanity if you give us money to buy a conference space in France than whatever you'd do with it" crowd -- the fact that it was a castle shouldn't be relevant.
Actually, the fact that people think castles are cool suggests that the going price for them is higher than their concrete utility would make it, since demand would be boosted by people who want a castle because it’s cool.
Did these guys have some special use case where it made sense, or did they think they were the only ones smart enough to see that it’s actually worth buying?
In other words, they investigated themselves and cleared themselves of any wrongdoing.
It was obvious at the time that they didn't need a 20 million dollar castle for a meeting space, let alone any other meeting space that large.
They also put the castle up for sale 2 years later to "use the proceeds from the sale to support high-impact charities" which was what they were supposed to be doing all along.
If I am donating money and you are buying a conference space on day 1 I'd want it to be filled with experienced ex-UN field type of people and Nobel peace prize winners.
Otherwise it looks like a grift.
The biggest downside, of course, is all your conferences are now in the same location