A Meta-Scientific Perspective on Thinking: Fast and Slow (2020)

>> In conclusion, Daniel Kahneman is a distinguished psychologist who has made valuable contributions to the study of human decision making. His work with Amos Tversky was recognized with a Nobel Memorial Prize in Economics (APA). It is surely interesting to read what he has to say about psychological topics that range from cognition to well-being. However, his thoughts are based on a scientific literature with shaky foundations. Like everybody else in 2011, Kahneman trusted individual studies to be robust and replicable because they presented a statistically significant result. In hindsight it is clear that this is not the case. Narrative literature reviews of individual studies reflect scientists’ intuitions (Fast Thinking, System 1) as much or more than empirical findings. Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologists, rather than an objective summary of scientific evidence. Moreover, ten years have passed and if Kahneman wrote a second edition, it would be very different from the first one. Chapters 3 and 4 would probably just be scrubbed from the book. But that is science. It does make progress, even if progress is often painfully slow in the softer sciences.

For example, I always had the suspicion (which may be wrong) that in questions like "which is a more likely outcome of 6 coin tosses, HHHHHH or HTHHTT", people tend to understand this question as: "which is a more likely outcome of 6 coin tosses, a sequence of 6 heads, or another sequence without any obvious regularity in it", in which case the latter is clearly the correct answer.

Now, in my experience, people with slight autistic tendencies often have problems with sorting their impressions into broad, general categories, but instead always analyze (even remember) the exact impression (or "special case") they are confronted with. For people like this, the question "HHHHHH or HTHHTT" is a completely different question than for the majority of the population, for which HTHHTT is a class of outcomes, not a specific one.

So, the problem to me does not seem to be a problem with human intuition for probability (or with "System 1", in Kahneman's words), but just one of semantics. If you go further and assume that scientists have a higher percentage of people with autistic tendencies than the general population, then the problem simply becomes one of communication between the researcher and the subject.

But since I'm battered with distrust thanks to the events in 2011 and later, I can't help but wonder if the replication index should be trusted.

The science is still emerging so of course its impossible to write an accurate book of this breadth, but does that make writing a book wrong? I don't think so. As the conclusion says it just needs to be "read as a subjective account".

I read "Thinking: Fast and Slow" something like 7 years ago now and while I wouldn't be able to give you a direct account of the ideas I've taken away from it at this point, I wonder how much partially or completely incorrect ideas on thinking I internalized from it.

Obviously I can now adjust my mental model but pop sci still feels a bit like a mine field for the quality of thought I may be taking away from it. I just started "The Selfish Gene" and this post has me wary of the quality of it even though just like Fast And Slow it's fairly critically acclaimed.

I think if anything in the last few years I've found myself being highly skeptical of most claims and ideas (which is also a balancing act, but largely good) which wasn't necessarily the case when I was reading Fast and Slow instead committing a common sin and taking the fact the author was a well regarded practitioner in their field at face value.

I guess the point of my comment here is I find that without taking (or really having the) the time to follow every idea to it's foundations and given there are an incredible number of ideas being thrown around openly and with little barrier to entry having to rely on some type of illogical filter and tendency towards your own biases is required just to filter down to a smaller set of ideas that at least seem plausible. The issue with that as we see on a day to day basis is that most people's filter (including my own given the critique of this book) sucks and we wind up with a bunch of plausible but incorrect ideas being treated as fact and propagated throughout the world. It would be nice if finding some kind of objective truth didn't require turning seeking that truth into a life mission.

It's also something to be said that most people operate on these faulty models throughout life to great success by conventional standards so it's not like knowing some set of objective truths, if they're there in the first place (I'm not entirely convinced everything has an objective truth), is a pre-requisite to a happy, successful or fulfilling life -- they may even be the opposite. I just have always found that unpalatable and hope we find ways to think better.

Yet, in other fields the standard for discovery is p=0.0000003.