My experience at and around MIRI and CFAR

I think that all it really takes to become extremely paranoid is listening to enough paranoid talk over a period of time. Some years ago I went through a period where I was on YouTube or something watching InfoWars-type content on a daily basis (and other conspiracy theorists). After a fair amount of this, I was quite afraid of the government. I know there were days when I was afraid someone was out to get me for literally no rational reason.

But that type of content is designed to make you feel that way. It's part of their weirdly effective hook. But at the core it's really just repeating a stream of extreme or paranoid perspectives in different ways.

Point being, I believe if someone decided they needed to convince a group of people about how corrupt the world is, or whatever, they could do so. One aspect of this might be a little bit of social isolation, just based on the fact that if you are in contact with fewer people then you are likely exposed to fewer information streams. So when they mention people who have very different ideologies, very high IQ, or anything else that might make it slightly harder to socialize, that seems to add up also.

The toughest part to me is that deep down I personally believe some core motivator of this paranoia IS rational in a way. Because my speculation is that we will see these general purpose AI technologies in our lifetime and there actually are groups who seem predisposed to misuse them or build the wrong type of system and that could be truly disasterous. I think that obsessing over these types of speculative existential risks though obviously is not a sustainable way to live for most people.

Now imagine your career and passion is thinking about thinking, making computers think, and making sure thinking computer's don't destroy the world.

"The explicit strategy for world-saving depended upon a team of highly moldable young people self-transforming into Elon Musks. For a long time, what was asked of me in order to maintain my funding was to “become a self-debugger” — become capable of working with my own mind effectively enough to be able to change it in the direction that would be useful for the project, which it was assumed was also the direction I’d obviously want, because right now I wasn’t a powerful Elon Musk type, and obviously I was here because I didn’t want to keep being an ineffective, unagentic powerless normie forever (this was not remotely the self-concept or set of concerns with which I entered Leverage, but it was a shameful prospect by the time I left)."

What?

"Perhaps more important to my subsequent decisions, the AI timelines shortening triggered an acceleration of social dynamics. MIRI became very secretive about research. Many researchers were working on secret projects, and I learned almost nothing about these. I and other researchers were told not to even ask each other about what others of us were working on, on the basis that if someone were working on a secret project, they may have to reveal this fact. Instead, we were supposed to discuss our projects with an executive, who could connect people working on similar projects. Someone in the community told me that for me to think AGI probably won't be developed soon, I must think I'm better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness."

What?

I tried reading the article, but it seemed like I had landed in some parallel universe, as I'd never heard of these outfits or people.

This is what organized religion eventually evolves into?