Trapped priors as a basic problem of rationality
astralcodexten.substack.com
astralcodexten.substack.com
But in fact many political zealots never accept reality. It's not just that they're inherently skeptical of what the other party says. It's that even when something is proven beyond a shadow of a doubt, they still won't believe it.
This could describe cognitive error [1] or it could describe someone who's using deontological reasoning more than consequentialist reasoning. (And perhaps lacking the vocabulary to say "consequentialist arguments won't sway my values.")
It is probably a good strategy to refuse to be talked into positions against your fundamental values by mere evidence. I'm joking but also serious. I have a vague sense that torturing suspects doesn't work to prevent terrorist attacks, but that's not the root of my opposition to torture. Maybe clever torturers could make it work more often than not. That wouldn't sway me, because torturing suspects is against my values. Documenting how well it can "work" won't make me reconsider.
[1] Further food for thought: "cognitive error" and "revealed preference" may also be different names for the same thing. Are they experimentally distinguishable?
As for the rest, yes, I agree with you. Often, after having considered a great deal of evidence, I condense the finding into some strength of belief about something. Now, absent the original evidence I have a position. This is usually fine since I actually lack the ability to summon all evidence on demand.
It's okay, though, since I (and I suspect most people intuitively) have both a likelihood notion and a certainty in likelihood notion analogous to a Confidence Interval and P-value, if you will.
So I might hold the belief that GME is going to be between 120 and infinity on Jan 13 2022 and have a certainty in that belief that is like 10% but lose the evidence that made me think that.
This is also sort of a failure mode for fast Aumann Agreement even in co-incentivized individuals - the fact that memory is short.
So the problem is that my certainty holding is not well done.
Thanks for the thought prompt.
Surely a rationalist would be into logic and actively refuse to read studies, like the Austrian economists did.
That sounds like an extraordinary claim that would require extraordinary evidence. No doubt physics and engineering have made great advances, but that doesn't mean that every social science that dresses itself up in the language of rationalism will be able to make similar advances, or really any advance at all. In fact, there is a case it will lead to regression rather than advancement:
Societies are very complex things made of people who respond to changing incentives. I don't believe anyone understands them very well, nor has any field in the social sciences established a track record of making highly accurate out of sample predictions, especially for the efficacy of interventions 10, 100, 1000 generations on. Even the simple question of "what keeps a society together?" is still a puzzling one, especially now that post-modern Western societies are disintegrating before our eyes and are failing to even biologically reproduce at replacement levels.
And so we turn to values. At least with values, it is possible to gain experimental knowledge in the laboratory of history. Societies reproduce values and compete with each other as well as with a hostile environment, so long lived societies should have acquired value systems more conducive to social survival.
And I don't think we can make much stronger claims than that.
Certainly someone just imagining stuff or doing some small experiments is unlikely to come up with robust rules for organizing society that will stand the test of time better than what comes from the long lived traditions of successful societies.
We sometime learn from evolved biology to make machines, but sometimes we can also see that evolution made a bad design. By learning from what has worked it's not too difficult to design an alternate society that will work just fine for decades or centuries.
Some societies survive by preying on others. Some are stable but prey on their own. I don't want a life in service to a society that evolved to be strong. I want a society that exists for the benefit of the people in it, and improves as we learn better ways to do things. (The current system does change but different structures could be much more adaptable.)
And, yes, one can use alternative schools of epistemology if one believes so. This group of people derive theirs significantly from Popper's line of "critical rationalism" which is different from "empiricism" (the names here are the ones that annoyed our cousin comment over there).
This school has certain affordances that permit them to reason as they do but if you disagree the differences between us are likely so large that it is unlikely that I can explain without expending far more effort than I am willing to.
So, for example, in the climate change example he gives, while it is of course appropriate to take into account possible biases of people promoting one position or another, and ways that those may result from the different values that the people with those biases, one should nonetheless endeavor to have a tendency to believe what is actually true about the factual matter, even if it were that the people with the wrong values were biased as an (indirect) result of those values, in a way that led them to be more likely to reach the correct conclusion about the empirical facts, than those with the good values.
(Not to say that I think this is the case in this example. Just, hypothetically. Or, what do you call something that is like a hypothetical but without expressing any position about whether the thing is actually true?)
In addition, while they may get tangled up and which is which may change for a given person over time (due to human thought being fuzzy and wobbly), I think it still makes sense to make a distinction between terminal values and instrumental values. Further, if one values something instrumentally, because one believes that it is instrumentally beneficial for one's terminal values, and it is demonstrably true that it actually does not benefit one's terminal values, then one should come to believe this truth, and so no longer instrumentally value the thing.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
People will sometimes understand their past behavior as arising from a cognitive error, and endeavor to address this. I suppose you could regard this as simply conflicting preferences, or preferences about preferences, but I don't think this is the best way to understand things.
That's true, and I have been convinced of some of those things. For example, I am empirically convinced that mass shootings by spree killers are shocking but not a major issue of public health or safety in the US. Even though I believe that Democratic politics gives disproportionate attention to mass shootings, weirdly mirroring how Republican politics gives disproportionate attention to Islamist terrorism, I'm not going to say so in the company of many Democrats. Because another thing I have been empirically convinced of is that people build effective political coalitions by emphasizing common ground and judiciously overlooking their allies' faults. I want action on climate issues, so I'm not going to alienate my natural allies by lecturing them about their irrational fear of semi-automatic rifles. Effective politics is almost the thematic opposite of a personal quest to aggressively probe all things in search of Truth.
I also don't think cognitive error can be entirely reduced in all cases to revealed preferences, because, uh, -- well, ok, one can, of course, model all behavior as arising from a preference to take exactly the sequence of actions that one takes. But I think you will agree that this is not a reasonable model.
It's a model that doesn't lend itself to mathematically grounded predictions. But is it actually not reasonable? Without access to another person's qualia it's really easy to misattribute whether their more surprising decisions are revealing a preference, demonstrating cognitive error, or servicing a fundamentally different value orientation. Like Hume's Problem of Induction I can't effectively grapple with it but I can't exactly dismiss it either.
I believe that too. There are some boxes so dangerous we need people not to open them even if they believe they should.
However there are a lot of things in the world that we can be wrong about. A strategy of "I'm going to use deontological reasoning" just means that we have to sort right from wrong by an evolutionary process over generations, getting to kids while they are young enough to be swayed, in a process that is a bit icky and disrespectful of people's beliefs. It would be better if people could be persuaded by evidence.
We don't get sewage, skyscrapers and subways by following our instincts or because some people somehow made them values. Everyone wants good sewage systems. The best parts of society come from following evidence and logic as best we are able - including, I point out, the legal system.
I don't think that you necessarily need to sway kids while they're young. 10 years ago there were no states in the US that had legalized recreational cannabis use for adults. Today there are 17, and I expect at least another 17 over the next decade. What changed? Attitudes shifted among all age groups. It didn't seem like a process of grandparents dying out and their grandchildren entering the voting booth.
It didn't seem like an evidence based process either. I was well persuaded in the 1990s that the deleterious health effects and other negative externalities of cannabis use were no worse than those corresponding to alcohol. The historical example of Prohibition was more than enough evidence for me that fueling a black market was worse than letting adults impair themselves. I wonder why evidence didn't persuade people to change laws sooner, yet the shift happened far faster than generational replacement once it started.
If anyone reading has a convincing model for how the cannabis attitude shift happened: does it only rationalize history, or does it also predict other things likely to change in the near future? There are a lot of ways we can turn past series of events into coherent stories. There are fewer that successfully foretell what happens in the next chapter.
A) Overincarceration of non-violent offenders
B) the reveal of racist roots in the foundational war on drugs rhetoric (which led to more strict penalties for the use of crack [cheaper street drug], for instance, than straight cocaine [high-society, white-collar drug]).
At least that seems to be the lynchpin from my point of view. It was always going to fail from mmy point of view since prohibitions in the U.S. don't have stellar records.
Religion is several of those boxes, which have already been opened for centuries.
Faith, (by which I mean deliberately attempting to believe something to a degree of certainty which exceeds what is warranted by the available evidence) is one of those boxes that should be sealed up and ejected into space, or at least made to be deemed immoral by the majority of people, since it obviously is immoral.
I believe that torture is wrong and I also believe that torture doesn't work. I could theoretically be convinced that torture works, but I would still believe that it is wrong. If I were to insist that torture doesn't work in the face of strong evidence to the contrary because of my moral beliefs that would be a cognitive error.
Regarding your footnote: They cannot be the same thing because preferences are prescriptive and errors are (at least as they are being discussed here) are descriptive.
"I would still be against torture no matter what the research shows" is revealed preference, and "I'm against it, therefore it doesn't work, no matter what the research shows" is cognitive error.
This is clearer if you suppose that I published a good, thorough study showing that torture works very well. In the former case, you could accept it as accurate but retain your opinion; in the latter case you would be forced to either change your stance or accuse me of [ineptitude, fraud, bias, secret sadism, etc].
“We’re an empire now, and when we act, we create our own reality. And while you’re studying that reality — judiciously, as you will — we’ll act again, creating other new realities, which you can study too, and that’s how things will sort out. We’re history’s actors . . . and you, all of you, will be left to just study what we do.”
- Karl Rove
A conspiracy theory is often something that can explain any lack of observational evidence for itself (because it's covered up!). If the prior on the conspiracy being true is too high, the update on contrary evidence will be very weak just because the conspiracy theory makes weak predictions about upcoming observations, so it loses probability mass very slowly as a whole, with probability mass mostly just being shifted within the context of the theory.
("They're covering up aliens!" --> no evidence of this after fifty years? --> "It's worse than I thought; they must be using mind control!")
The only cure for this might be preventative -- just maintaining a strong doubt on deep conspiracy theories. But that also means resigning yourself to never believing in one before it's revealed. If it turns out you live in a reality where the government is using mind control to conceal aliens, you'll not be the one who realizes it before everyone else.
That may be the case for conspiracy theories, but I'd expect the theory that dogs are dangerous to make much stronger predictions. On the other hand, maybe people who're afraid of dogs don't expect them to behave differently, they just think that everything dogs do is scary (which is true, since they're scared by everything dogs do.)
In this example, if their fear of dogs causes them to misperceive a dog's excitement as anger, then it's closer to the scenario in the article. But if their paranoia is such that they perceive the dog is acting friendly but interpret that as the dog luring them into a sense of safety to bite them later, then it's more like the conspiracy theory case.
> I've had arguments with people who believe that no pro-life conservative really cares about fetuses, they just want to punish women for being sluts by denying them control over their bodies. And I've had arguments with people who believe that no pro-lockdown liberal really cares about COVID deaths, they just like the government being able to force people to wear masks as a sign of submission. Once you're at the point where all these things sound plausible, you are doomed.
Apparently you’re doomed if you believe one of those things describes reality more than the other? They’re both extreme generalizations, obviously, and are defined imprecisely (“no true Scotsman”), but couldn’t it be that one of those two groups of people have beliefs that correspond much more with reality? Or is the conclusion that, since this is politics, and political views is intensely personal and prone to bias, that both sides are equally correct?
Isn’t it possible that there actually are some powerful political groups who have honest and good intentions and others which do not? Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group? Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct? Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
If your faith in a flat earth is strong enough that you'll come up with new theories of optics to justify continuing to believe in it even after you're launched into orbit to see it firsthand, your belief is unshakable. But that doesn't guarantee that you're wrong, maybe you're a genius who just revolutionized cosmology and optics. But you're probably wrong.
Yeah, that’s the kind of epistemological meat we really need to slice into. My view is that certainty is impossible but that knowledge is possible, and that we get ourselves into a lot of messes when we confuse knowledge with certainty. The quest for knowledge should not be a quest for certainty (either in the emotional sense or the Bayesian sense) or a quest for a perfect method of obtaining or justifying knowledge.
But...isn’t that statement either true or false, just like the statement “the Earth is flat” is either true or false? What’s the difference between the two statements? Is knowledge about one statement possible, but not the other?
If you believe "flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth" , this would pose difficulty for changing the belief "the earth isn't flat".
If someone believed "people who claim that 'flat-Earth advocates don't actually care about whether the Earth is flat or not, they just want to prevent space exploration to keep us trapped on Earth' don't actually care about whether flat-Earth advocates really [...], they just [idk, some absurd motivation for making the claim] ", that would pose difficulty for changing the belief "flat-Earth advocates really do believe that the Earth is flat".
I’m not sure why. The normal simple tests to distinguish between a flat Earth and a ball-shaped Earth ought to still work independent of what flat-Earth advocates believe.
The statements being made aren't about the earth, but about people's perception of the earth. And ultimately you can't objectively determine someone else's qualia.
What GP is suggesting is that if the claimed flat-earth belief is simply a justification for a different goal (prevent space exploration), then providing contrary evidence doesn't matter, because the belief isn't core of the issue. But you can't really know that independently. And this is especially true where you get a sort of metacircularity in beliefs and belief systems. Identifying what the "real" axiom someone subscribes to is difficult.
To look at the conspiracy theory about flat earthers that GP proposed, imagine that tomorrow, suddenly, the Earth was flat. All of the historical record was the same. We have evidence and recordings that it was round, and that the Greeks agreed, but tomorrow we couldn't reproduce those experiments. The horizon suddenly looked different. A reasonable conclusion here might be that we're all suffering from a collective delusion (and that in fact the earth is still round[0]), because our prior that the earth could rearrange itself overnight is low. But, this person might believe that the collective delusion is the result of machinations of the flat earthers, trying to trick us all into stopping further space exploration by convincing us that the earth is flat. This despite there being no evidence that flat earthers were at fault, or even capable, of causing such a worldwide event.
[0]: This actually objectively doesn't make a whole lot of sense, if we're capable of being subject to a massive collective delusion at worldwide scale, it could be that we were doing so before, and that the change is the delusion wearing off, but I digress.
I don’t think there’s some process to follow to acquire knowledge with a guarantee that we haven’t made a mistake, but I don’t think that means it’s impossible to acquire knowledge. I think we can and do acquire knowledge about people’s beliefs and intentions and use that knowledge to solve problems. Obviously examples are convicting people of fraud and various crimes where the person’s intentions are relevant.
I'm not saying reliably discovering intentions is always impossible. I'm saying that it is not always possible.
Of course, Scott could be wrong about this thesis. It's possible that maybe believing that pro-life conservatives don't care at all about fetuses doesn't actually indicate that you have a trapped prior that will prevent you from ever hearing new arguments on the abortion issue. But that is the argument that Scott is advancing, not that you can't form opinions about the truth or falsehood of someone's political views at all.
That doesn't mean there are some people who have no bias at all. Just different ones that apply to other things.
In the examples you quoted, he was deliberately listing views extreme enough to be inaccurate. For example, both the liberal and conservative examples said the characterization applied to _everyone_ in those groups. The less-biased interpretations might be: _some_ conservatives really _do_ care about fetuses and lives, and _some_ liberals _do_ care about preventing COVID deaths.
I think it's important to really carefully spell out what we mean by "not having good intentions". Because there are two different reasonable things that that can mean, and they're _really_ different, but the incentive to flip-flop between them can be strong.
The first meaning sounds like "having bad ideas". For example, let's say I'm in favor of some new law to...promote the arts. (Trying to pick a bland example.) You might think my law is actually going to hurt the arts, and argue against it for that reason. Or maybe you agree that it'll help the arts, but you think its other downsides outweigh its upsides. At maximum generality, maybe you don't object to my idea per se, but you'd just rather spend the same money on something else.
The second meaning sounds like "having bad values". For example, you could claim that I'm not actually interested in art, and just trying to divert money to my friends. Or maybe I could paint you as some sort of uncultured caveman, who can't understand why normal people like art.
When we put it that way, I think it's clear that we usually say "good intentions" to talk about the second meaning. Like, the whole point of saying it is usually to _contrast_ someone's good values with some bad outcome they're associated with, right?
On the other hand, it's much easier to take an argumentative position using the first meaning. The world is complicated, everything has downsides, etc. Plus object level arguments tend to be more sophisticated and interesting, with expertise and fancy math getting involved. So a tempting strategy can be to equivocate between "having bad ideas" and "having bad values", by making an argument about ideas or outcomes and yet drawing or implying a conclusion about values. This is the sort of fallacy that internet rationalists complain about constantly, calling it a "motte and bailey" or something like that. And I'm really sympathetic to that complaint: I think that it's very easy to make this sort of bad argument, that it's actually kind of hard _not_ to make it without some sort of shared community or commitment, and that the effect of this sort of argument on discourse is frustrating, unproductive, and ultimately dehumanizing.
So all that said, do I think that some groups objectively have bad values? I...oof...I feel kind of forced to admit that it's true sometimes. I can't sustain a really extreme position here like "everyone's values are special and lovely in their own way", at least not without diluting the concept of values so much that it's not very interesting anymore. But at the same time, I just feel...super skeptical of most "bad values" arguments in practice.
Two people may have different weights on the proper tradeoffs of safety versus liberty, and each might think the other has "bad values" without disagreeing with the others ideas, or thinking they are being dishonest or underhanded in any way.
There are people who have bad values, and want things that are bad, yes. (Though generally they don't want them because they think they are bad.)
> Isn’t that possible even if most people have a tendency to believe that their own groups are the good ones, and that belief is just as strong for each group?
yes.
> Is it impossible for me to believe my group’s intentions are honest and good, and you believe just as strongly that your group’s intentions are honest and good, but that only one of us is correct?
I'm not sure what precisely you mean by intentions being "honest and good". If I believe that my intent is honest -- well, I suppose I could be fooling myself, thinking that some behavior of mine is "honest" by twisting the meaning of the word "honest" in my mind, while still intending to cause others to have a belief which I believe to be false. Also, by intentions being "good", do you mean that the things one intends are actually good things, or that the one intending them thinks them good for those they impact, and intends them for that reason?
> Is knowledge nothing more than a feeling of certainty? Or is knowledge simply impossible?
Knowledge is more than a feeling of certainty, yes. And, either knowledge, or something very similar to knowledge, is possible.
For example, both of the beliefs quoted, are false. There may be related claims which are true, but neither of the claims, as stated, are. (I don't see the no true Scotsman aspect of either though?)
The point isn't that different sides are equally correct. The point is demonstrating a kind of error that can be made, and giving an example on multiple sides makes it easier to understand. If one only gave examples of a type of error being made by the side the reader agrees with, there would be a danger of this being interpreted as simply an attack on the side the reader agrees with, and so the reader would be less likely to appreciate the point. On the other hand, if one only gave examples of the type of error being made by the side the reader disagrees with, while the reader may take a more positive view of what is being written, they are less likely to internalize the point as a kind of error that they, or others on their side, are prone to.
For something to be a general type of error, the type of error should be one that can be made in multiple contexts/directions/ways.
Then, to demonstrate that something is a general type of error, why not demonstrate that, by showing that it can be made in multiple contexts/directions/ways ?
The point isn't nihilism or relativism. The point is vigilance.
It could mean something objectively moral, or if you’re not into that sort of thing, it could just mean intentions that both groups in question would agree are good.
I do believe that at least many moral questions have an objective/real/true answer, and maybe all well-formed ones do. (And I’m confident that at least almost all of the ones that are relevant in practice have a true answer.)
Ok, so you meant that the intent is for something which is good.
Then yes, people can have an intent for something which they think is good, and have their intent for it be because they think it is good, while they are incorrect, and in truth the thing they intend is bad.
The reason I was unsure is because I think there’s an idiom or something(?) where “good intentions” vs “ill intentions” refers to whether they intend to do what they think is beneficial to the target, vs to do something they think is against the target.
Thanks again for clarifying .
Don't get me wrong. I took statistics in college, and we covered Bayesian methods. They can't be controversial because Bayes' Theorem is a theorem, meaning it's proven. But the attempt to apply its terminology to lengthy but "soft" arguments seems like pure clutter.
Nevermind that the first principles lose all context of the original issue - we're in a world of pure logic, where we can use theorems! To the rationalist community, everything looks like a good opportunity to write a lengthy essay reinventing the wheel with Bayes theorem. It's a cargo cult of basic statistics dressing up otherwise uninteresting arguments.
It's very tedious for people like myself who actually studied statistics and apply it professionally. Bayes theorem is barely even a theorem - it's literally a basic rearrangement of the axioms of probability. It's not some supreme revelation that imbues arguments with more credibility. Most of the time I see it invoked in discussion, it's not even rigorously quantified. You can choose arbitrary priors, so at the end of the day you never arrive at an objective truth. It's profundity for its own sake, and nothing is really achieved.
It is also certainly true that garbage in and garbage out, i.e. if you feed garbage inputs, you'll get garbage out (Bayesian or not), but the advantage in my opinion of having a framework for making decisions in cases of uncertainties is extremely useful. It is often hard to really apply this for ordinary life situations, (and maybe often not worth it), but even then it's useful to think about priors/evidence, even if you don't particularly care about the final number.
With the very important proviso that the events you're uncertain about are at least probabilistically quantifiable and the "priors" are credible. It's very easy to write a long winded essay arriving at any conclusion you want if it's a subject with significant uncertainty and you can make a halfway decent appeal to whatever priors you want.
The second is there's no way to set priors for silly ideas and they won't admit some things just ain't gonna happen, which means they've accidentally started a religion by refusing to not believe in various logic traps they call "existential risks" like Roko's basilisk, evil AI gods enslaving them, and so on.
* for some reason they call themselves rationalist which implies being "rational", aka "always right", even though they are explicitly not rationalist, but empiricist!
Being able to think about probabilities is useful, if they're quantifiable. And conditional probability plays a useful role in that process.
> The other promising source of hope is psychedelics.
For me, what this part if the essay points to is that everyone needs to do their own work, in whatever context, and there's no getting around that.
Experts, authorities, and managers need to find the line between facilitating a process and doing the process themselves. You can't make anyone believe anything or do anything. You can only hope to facilitate the conditions where helpful beliefs and accomplishments arise.
I hope this is one significant cultural shift during my lifetime. The 20th century was about creating expertise and giving that expertise authority, which is just another expression of the dominator paradigm.
Edit: I found it with a less-quick search; it's called D-cycloserine.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4081005/
I think there is still other research on using other memory-affecting and/or emotion-affecting medications to facilitate exposure therapy.
a Bayesian updater takes in new evidence, and the adjusts their priors based on that evidence. Crucially, the information that is fed back into the Bayesian machine should be independent of the prior.
People do not have direct access to evidence. They only have their perception of that evidence, and the content of those perceptions depend in various complicated ways on people's priors. Therefore, evidence that should move people's priors may move them less than they should.
Sounds plausible. I think reinforcement learning algorithms like temporal difference learning are probably a more accurate description of how people learn tha Bayesian update, but the issue described would apply regardless.
And I think that everyone has a worldview. No one is impervious to this or truly non-partisan. Some people may be less partisan in some ways. But they still have worldviews that influence their thinking, possibly in somewhat different areas.
Edit:
Less snarky response here. The “just-so” style that the author uses to explain psychological phenomena appeals to the readers’ cognitive biases and avoids confronting the reader with the more rigorous, mechanistic (perhaps evidence-based?) explanations that the same author proposes as a way to combat said cognitive biases. This strikes me as mildly ironic.
This is endemic to the rationalist community. It's actually quite humorous once you begin to notice it.