Yudkowsky doesn't strike me as a crank. Why do you say that?
He is a minor internet celebrity whose only claim to fame is writing fanfiction about AI, the whole "rationality" cult, and is a self-proclaimed expert on matters where he shows no achievements (like AI) while making unsupported doomsday predictions about evil AI, the Singularity, etc. Also, that deal with Roko's Basilisk that he now wishes never occurred (oh, yes, "it was a joke").
Mostly, someone with no studies and no achievements making wild doomsday predictions. Doesn't that strike you as a crank?
An analogy would be if I made wild assertions about the physics of the universe without having studied physics, without any lab work, without engaging in discussion with qualified experts, with no peer reviews, and all I presented as evidence of my revolutionary findings about the universe was some fanfiction in my blog. Oh, and I created the Extraordinary Physics Institute.
If anything, I'm surprised that this philosophy isn't mentioned more in a thread where the author gleefully talks about ML being used to create better AI, layer by layer until the thing is even more opaque than what we're currently working with.
This is terrifying, as we currently have only very loose ideas about how to reliably ensure that a powerful reinforcement learning system doesn't accidentally optimize for something we don't want. The current paradigm is "turn it off", which works well for now but seems like a fragile shield long-term.
At least inasmuch as anyone who thinks about stuff can be considered a philosopher. But he strikes me much more as a self-appointed expert on matters where he shows no achievements.
He writes fanfiction about AI rather than actually doing stuff with AI.
I'm arguing that what he writes is fanfiction (which he takes way too seriously), and that he's not an expert in AI and therefore we shouldn't take his predictions too seriously.
That's a pretty weak argument. I love scifi, and I love for example Philip Dick's writing, yet I would not consider PKD's opinion on the future of AI/AGI particularly relevant.
James Cameron is not an authority on AI either.
If people said "Yudkowsky is a nice fanfiction author" it would be one thing. But he considers himself an actual AI researcher, and that's just not right. He is not qualified, and has no accomplishments in the area, other than writing fanfiction about it.
Here's a sampling of his non-fiction writing:
https://www.lesswrong.com/tag/sequences
https://intelligence.org/files/EthicsofAI.pdf
https://arxiv.org/abs/1710.05060
https://intelligence.org/files/AlignmentMachineLearning.pdf
https://arxiv.org/abs/1401.5577
https://arxiv.org/abs/1401.5577
https://intelligence.org/files/Corrigibility.pdf
https://intelligence.org/files/DefinabilityTruthDraft.pdf
https://intelligence.org/files/IEM.pdf
https://intelligence.org/files/TilingAgentsDraft.pdf
But you'll just say there's nothing of value there, and it's somehow figuratively "fan fiction", because he didn't go to college, and he doesn't work much on ML, which is clearly the end-all of AI.
Science fiction writers tell entertaining lies to amuse their readership. They are generally not really trying to "pain correct visions" of anything, and you are greatly exaggerating the extent they have success in this endeavor, which again, is generally not something they are even trying to accomplish.
I think it's a net good when philosophers mentally explore scenarios most people don't consider, regardless of their plausibility, but they will not be taken more seriously that Peter Watts looking for inspiration for his next scifi book unless they offer some kind of evidence to sustain their conclusions.
The handful of philosophers I've known are very smart, interesting people, but except for one who teaches philosophy, they don't live off philosophy. Most have a side job, and one lives with his parents. It's as if philosophy was more of a mentally rewarding hobby than a job.
I have no doubt that he was thought a crank by many, nor that many other actual cranks were claiming discoveries based on Maxwell's work. One can only tell the difference in the rear-view mirror.
Talking about paradigm shifts again, not a single one of them matched the consensus at the time.
Isaac Newton spent a lot of time thinking about alchemy and religion, for heaven’s sake. During his lifetime, it wasn’t obvious even to the smartest thinkers whether science, the Bible or quasi-religious rituals was the best tool for understanding the world.
Conventional thinkers expect that the future will look like what they know, and it leads to frequent, overconfident dismissal of everything that is unconventional.
By all means, disagree and explore in your own direction. But don’t go around degrading those you disagree with. It’s just so average.
The word "crank" has the connotation of someone self-deluded, aggressively promoting their beliefs and reacting badly to critique, and who decides for some reason to ignore the normal channels of peer-reviews, academia, and scientific research.
You can be wrong, and then you can be wrong and also lack formal education, forgo presenting your findings in peer-reviewed journals in the relevant area of your research, and decide that instead of joining the mainstream, you can completely sidestep it by creating your own "research institute" (of which you are of course a "fellow", because why not). Your findings and papers can then be self-published on the internet, bypassing any quality controls. Bonus point if your theories paint a fringe doomsday picture, "Roko's Basilisk is out to get you", "the most important existential threat to humanity is malign AI", etc.
About the only item in the crackpot index that Yudkowsky doesn't tick is the "they are trying to suppress my truth!", to his credit.
Each of those can be of little importance, but all taken together paint a pretty definitive picture.
Unreservedly agree. Even if they're not as fruitful as Oliver Heaviside's effort, a mentally rewarding hobby is ultimately your business, and as long as you don't harm others, no one has grounds to judge you.
If Oliver Heaviside had started giving conferences warning that Maxwell's equations demonstrate FTL communication is possible, which could trigger a paradox that collapses reality, he wouldn't have such a good view in the rear-view mirror unless he offered some kind of evidence. Maybe Oliver would have been right, maybe not, it would be impossible to judge without concrete evidence.
If the latter, anybody can do that.
(Also on the tech practicioner side there's obviously lots of major figures that don't have formal qualifications)
If someone is going to market themselves as an AI expert, I expect them to have experience and qualifications to back up their opinions.