Damn, there's a lot of nonsense in that essay.
"Rationalists believed that Bayesian statistics and decision theory could de-bias human thinking and model the behaviour of godlike intelligences"
Lol
"Rationalists believed that Bayesian statistics and decision theory could de-bias human thinking and model the behaviour of godlike intelligences"
Lol
The reasoning improvements primarily cone from a greater awareness of words (ambiguous meaning, context sensitivity, considering things as if they were statements in set theory) and tools like word tabooing or outside view.
I doubt the author has had much contact with rationalists (who the hell says "rationslism"?!).
And if they fail to do this, all life on earth will soon be extinguished (maybe in the next few years, probably not more than 50 years), and potentially not just destroyed but converted into an unspeakable hell.
An offshoot of the group started protesting the main group for not doing enough to stop the apocalypse, but has been set back a bit by the attempted murder and felony-murder charges they're dealing with. Both the protests and the murder made the news a bit but the media hasn't managed to note the connections between the events or to AI doomer cult yet.
I worry that it's going to evolve to outright terrorist attacks, esp with their community moralizing the use of nuclear weapons to shut down GPU clusters... but even it doesn't its still harming people by convincing them that they likely have no future and that they're murdering trillions of future humans by not doing everything they can to stop the doom and by influencing product and policy discussions in weird ways.
[1] I think, this is Yudkowsky, Lesswrong, MIRI, the doom cult it became, the crazy revelations like https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experie... and https://old.reddit.com/r/slatestarcodex/comments/qa14kg/my_e...
Did I miss some news?
https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
The gung-ho crowd will always look like crazy types to the outsiders
Are you talking about anyone but the Ziz cult? They're very far removed from typical rationalist sphere think. Just because a schizophrenic psychopath reads a blog and spends all their time trying to manipulate other vulnerable isolated readers doesn't mean it's because of the bigger group they were trying to associate with. Those people are homeless self-proclaimed "vegan siths," hardly the employed MD / graduate types that ACX / lesswrong / EA is made of. If I go do something terrible, you and HN are hardly to blame.
And yeah, they're extreme, but also made them more vulnerable to it and more likely to act out. (And also, thus, more likely to form a plot to murder their landlord)
When some psycho goes to an AI conference with a bomb vest on and their views are directly traceable to EA and LW then it absolutely will be their fault as it is the expected outcome from years of brainwashing that AI will bring about the end of the world and potentially eternal torment. It's the expected outcome from saying that using violence to stop it would be morally justified (which is, incidentally, also inherent to the AI God story: the idea is that it will forcibly prevent the creation of other super intelligences, and not by talking people out of it, it's always been a power and control fantasy).
It's unlikely to be the people at the top that will be pulling the trigger, they're going to be too busy enjoying the traditional fringe benefits of having a cult. It's the people at the bottom that haven't figured out that the purpose of any cult is locking other people into extreme views so that the people at the top of the power structure can control them and profit from them, so they can enjoy relative fame and having their interests catered to... the people who are marginal and at the edge, not too valuable to the cult themselves but electrified by its message.
It'll be that marginal member who's group approved drug use wasn't kept under control, who didn't find success in the cult group sex harem, etc. It'll always be the fault of the person who pulled the trigger, but also the fault of the people who put their finger on it.
Do you actually have a definition of "brainwashing" that isn't equivalent to "convincing" here?
I really shouldn't engage with you because you seem far more interested in the emotive connotations of words than the actual meanings of them but I'm admittedly a bit curious if you are aware of what you're doing.
(You clearly don't have any actual knowledge on the subject if you think there's any profit involved here, I know a lot of rationalists, our net donations to MIRI is $0. Various anti-malaria causes? Likely over a million)
They have made posts (well, I can remember at least one, which I think was well-received) about how people working on AI safety/alignment should avoid having “leaving their fingerprints on the future”, meaning, making the AI aligned to specifically their values, rather than an impartial one.
So, I think they generally believe they oughtn’t have it aligned specifically to their values, but that instead they should try to make it aligned with a more impartial aggregate of human values. (Though they might imagine that a “coherent extrapolation” of typical human values, will be pretty close to their own. But, I suppose most people think that their values are the correct ones, or something else playing the role of “correct” for the people who don’t think there are “correct values”.)
Do they put enough of an emphasis on this? I’m not sure. I can say that I certainly don’t look forwards to the development of such a superintelligence (partially on account of how my values differ from theirs and from what I imagine to be “the average”). But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”.
So, I’m mostly hoping that either superintelligence won’t be developed, or that if it is, God intervenes and prevents it from causing too much of a problem.
> But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”.
I'm not sure about that. Part of the capability to do great harm is from the near miss. Fighting off Mindless grey good is better than some techbro trying to "optimize" you against your will.
Of course this line of thinking requires accepting their inane premises, but it's worrysome that even if they were right the astonishing hubris would make them the existential threat more than anyone else.
I think the greater relevance is the underlying unhealthy mental states underlying the doom cult. A person living a life in service of a very fringe idea we'll all likely suffer ultimate doom beyond measure unless their social group gets its way is going to exhibit all manner of compromised judgement.
Good luck fighting off something smarter than you. Has worked out great for the leopards.
The AI X-risk statements are public.
> has been set back a bit by the attempted murder and felony-murder charges they're dealing with
You have no idea what you're talking about.