Hacker News Reader

Rationalists generally prescribe a 'Bayesian' world-view, which can extract useful information out of a chaotic, too-complex-to-model world.

Regarding the power of intelligence, it's generally considered to be synonymous with optimization in the rationalist crowd. It's not about being all geeky and axiomatic, but using all information and tools available for optimization towards outcomes one wants

> I feel that the main issue with the rationalist crowd is that they live too much in the space of rationality, intelligence and abstractions, but not enough in reality.

This seems like your idea of what the rationalist crowd is rather than what they actually are.

It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.

So I must ask: what is your evidence/basis for these claims?

The Zizians would be a good start. As it turns out your sense of reality can be pretty malleable when living inside an echo chamber

If more than 0.05% of the rationalist and AI safety community were Zizians this argument might be worth considering for like ten seconds before dismissing it.

Every major political party has violent off-shoots, as does every major religion. If we're writing off rationalists because of a half-dozen people, we're writing off the vast majority of the world for participating in even worse guilt-by-association.

> It would be highly irrational to deny or ignore reality, including the influence of emotions, irrational humans, chaotic systems, etc.

Well, yes, but being a rationalist does not make one rational. It just makes you part of a group, and you show membership to such a group by applying a very specific brand of rationality: using the "right" words, the "right" ideas, the "right" way. A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.

> So I must ask: what is your evidence/basis for these claims?

It's not a monolithic community, so you'll see more debate and disagreement than in a lot of other groups. So what they "actually are" is many things.

But there's often a certain ungrounded "vibe" to the conversation there. It's a breeding ground for ideas and thought experiments that I would generously qualify as dubious. Stuff like Roko's Basilisk, Pascal's mugging, AI boxing roleplay, precommitment, time loops, whole universe simulation, recursive self-improvement, a superintelligence converting the entire universe to paperclips. One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).

It's the kind of thinking that makes the most sense in a smooth mathematical vision of the world, because mathematical objects are the kind that admit exponentials, extrapolations and infinities. But if you work with physical reality enough it becomes clear that it's a hopelessly messy thing that will never abide by your best laid plans. When e.g. you ponder how a superintelligent AI could escape from containment by simulating the gatekeeper's mind and say precisely what would make them free it, or threaten to torture a thousand copies of the gatekeeper unless freed, part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment. I think a lot of rationalists either lack this mental "sanity check" or don't trust it, which lets their mind drift into a weird space.

> A lot of people fetishize cold, hard logic and would rather hold all emotion in contempt than do the work of understanding why it exists and what purpose it serves.

In the rationalist community we're discussing? Is that what you're claiming here?

> But there's often a certain ungrounded "vibe" to the conversation there.

This really sounds like evidence for my claim, that what you're saying is based on your feelings towards that community.

> One of the community's most well-known outputs is a 1000-page Harry Potter fanfiction which I can only describe as a fantasy of solving everything with big brains (it's weird, but it's fiction, so whatever floats your boat).

The point of that story is to make technical concepts more accessible and easier to ingest [0], not necessarily to make a point by itself. Framing it as a weird fantasy is disingenuous.

> part of you is going to clock that as weird nonsense even though you are not able to explain precisely what's wrong with the thought experiment.

This is again, feelings presented as evidence. The fallacy of appealing to emotion ("It must be false because it feels wrong"). Thought experiments are primarily meant to provoke thought, not as reliable predictions of reality. People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.

[0] https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_o...

> This is again, feelings presented as evidence.

It's not presented as evidence. I was going for the "basis" part of your ask for evidence/basis, which is definitionally looser: elaborating on my thoughts and the kind of content I've seen that made me think this.

Let me put it this way: I have spent hundred of hours reading rationalist and rationalist-adjacent posts on LessWrong, SSC, HN, Reddit; the entirety of HPMOR, a decade ago; books like Superintelligence, The Age of Em; the LessWrong IRC channel, a while back; I have reasoned arguments about pretty much all of it, including the stuff I consider to be ungrounded. What I don't have is an entire day to collect the material and lay it all out to you in a 5000 word dissertation. Do you see what I mean?

So, yeah, it's a "feeling" -- it's the "feeling" I ended up with after spending all that time engaging with the material, thinking and arguing with people, which is why I trust it -- I have worked very hard to develop this intuition. And perhaps I could pull from it to craft an argument you would find compelling, with all the necessary receipts... but you must understand that this would be a massive time sink and I'm not going to do it. It's not a reasonable ask in the context of an HN thread. To a point you just have to trust me, or not trust me on this. I don't expect you to. It's fine! Not every thread can have a conclusion.

> People are free to indicate where any thought experiment is lacking using well reasoned arguments. "It feels wrong" can be a great start for that, but never a good end.

It's triage. The better your intuition is about whether an idea will lead somewhere, the less time you will spend pursuing dead ends. In an ideal world, you dismiss thought experiments will well-reasoned arguments, but the reality is that there is an opportunity cost to it. Using "feelings" to end some conversations is unfortunately a pragmatic necessity.

> To a point you just have to trust me, or not trust me on this. I don't expect you to. It's fine! Not every thread can have a conclusion.

It's clear how you got to your opinions. I see no reason why anyone that did not already share them would do after your comments, though. Some examples or bits of evidence would have worked towards convincing readers.

> but the reality is that there is an opportunity cost to it. Using "feelings" to end some conversations is unfortunately a pragmatic necessity.

The latter is something completely different than dismissing thought experiments based on feelings after having "pondered" and tried to rationally dismiss them. One of the key tenets of rationalism is exactly that feelings and intuition can be incredibly misleading. It's fine to have them and see value in them, but thinking they are reliable when you cannot rationally support them is nothing more than gambling.

I agree, and it's a particular shame in this instance, because what is startling about the HF incident - to me, anyway - isn't the degree of intelligence the agents exhibited but their persistence. I tend to believe that LLM architecture is not capable of producing a "superintelligence" in the way the LessWrong crowd defines that concept, but the combination of infinite stamina and infinite persistence is enough to cause some serious problems, even if intelligence plateaus right now.

When the "LessWrong crowd" talks about dangers of AI, they don't assume a particular form of intelligence or method to achieve it. They talk about the danger of optimization processes, i.e. "find X which minimize Y(X)" itself can be dangerous, even more so if X is a sequence of actions. "infinite stamina and infinite persistence" is one of possible forms of superintelligence in Bostrom's _Superintelligence_.

Yea, this is a pretty common failure mode for people.

Failing person: "You said that X could happen and it can't".

LW poster: "I said X, Y, and Z are likely possible. X turned out not to be true, but Y and Z are happening now".

Failing person: "No, you got one thing wrong so I can't listen to anything you say".

in a way you could criticize them as being idealist instead of materialist. like you said they think ideas are more important than material substrates.

most of the time brilliant human ideas are arrived at near simultaneously by multiple people because thats the affordance of technology and social and scientific development

its true ai will have better working memory and will have read more books than a person, but i don't think thats insurmountable at the frontier, which will develop slower than pure thought, since real materials need to be moved and manipulated. whoever holds the guns holds the power. the danger is giving ai effective cotrol of industrial and security processes, then it can fuck things up

thats the stuff of revolutions. when production was controlled by capitalists more than lords, the lords got overthrown. in many countries peasants and workers overthrew capitalists because they control production. hence much effort in us buisness goes into repressing revolutions. ai will be a vector the business people give control because they think it is more friendly to their interests than human workers, but they may lose either way.

To a degree you might say that they live too much in the space of the abstract and not enough in the concrete, or too much in the general and not enough in the specific.

I think that's a fair criticism a lot of the time. Most of the time we have precedent that can guide our actions well. Trying to reason everything out from first principles can be wasteful navel gazing when we've collectively seen the movie a million times.

Focusing on reality and specifics though is what causes people to say there's no global warming because December is cold.

It is sometimes possible to see a trajectory that has never happened before though and that's when you need the autists.