It’s still very much worth reading in its own right, but now implicitly comes bundled with a game I like to call “calibrate yourself on the replication crisis”. Playing is simple: every time the book mentions a surprising result, try to guess whether it replicated. Then search online to see if you got it right.
You can ignore anything said in chapter 4 about priming for example.
See https://replicationindex.com/2020/12/30/a-meta-scientific-pe... for more.
This, IMO, is the actual biggest problem with LLMs training on whatever the biggest text corpus us that's available: they don't account for the fact that not all text is equally worthy of next-token-predicting. This problem is completely solvable, almost trivially so, but I haven't seen anyone publicly describe a (scaled, in production) solution yet.
Can you explain your solution?
It's probably the AI thing I'm most excited about, and I suspect we're not far from that, although I'm betting the copyright battles are the primary obstacle to such a future at this point.
People from Europe, came to have the Industrial Revolution at just the correct moment.
Some small changes in history and it would have happened in India.
It is making a theory to fit the facts.
I do not think the author is a "white supremacist" but the book reads like that. Taking all the accidents of history and making them seem like destiny that Europeans rule the world (they do not, they never did, and they are fading from world domination fast)
Thinking Fast & Slow was a fun read, but I did not retain much more than the basic System I/II concept which I find is a useful device.
It's not just that individual studies have failed to replicate. The whole field was in question at least a decade before the book was written, and since then many of the foundational results have failed to replicate. The book was in a sense the last hurrah of a theory that was on its way out, and the replication crisis administered the coup de grace IMO.
I want something similar but for children's literature. From Ralph and the Motorcycle to Peter Pan, a lot of stuff doesn't hold up.
The books provide plenty of utility. But many things don't hold up to modern thinking. LLMs provide the opportunity to embrace classic content while throwing off the need to improvise as one parses outmoded ideas.
You could not redact piece of art out of "old ideas". It is like re-drawing classics paintings but mask nipples and removing blood.
And books which could be redacted this way without falling apart — well, don't read such books at all and don't feed them to the children.
Literature for children must not be dumbed down, but exactly as for adults, but better.
Also please omit "who has a better story than bran"
It's saying that the author's invented metric may indicate that the studies within each chapter may not replicate. No actual replication studies were done to produce the table in that post.
I included that blog post as a guide to what other results may be suspect.
Long before the controversy was public I was reading that book and, despite claims that the reader must believe the findings, it sounded like nonsense to me. So I looked up the original paper to see what the experiment set up was, and it was unquestionably a ridiculous conclusion to draw from a deeply flawed experiment.
I still never understood how that chapter got through without anyone else having the same reaction I did.
As he's shifted from primarily a journalist to primarily a storyteller, he's chosen to sacrifice additional information and accuracy in lieu of telling a consistent and compelling narrative that support what he thinks is the important thing to take away, not necessarily what you would take away were you to review all the same information yourself.
Under that understanding, I find him fun to listen to. The things he "reports" on/illuminates are interesting, but at this point I don't assume he's giving them an even handed representation, so his conclusions are not necessarily my own, and at best it's a set of things to look into and research myself if I find my interest piqued after a fun or interesting story is told.
If the best evidence that Gladwell could find to support his points, don't support his points, then it really makes me question the utility of trying to find evidence to support Gladwell's points.
It may be that, the reason why no subject matter expert makes the points that Gladwell does is because Gladwell's points are either wrong or not even wrong.
On the other hand if you see Gladwell for what I think he is, which is an entertaining storyteller using science as a backdrop to present his view of the world, and willing to twist it to suit his view, then he's really no different than many other writers. In that case you can enjoy him or not, at your leisure, just as you would any other editorial or op-ed piece. Entertainment is entertainment, and his content is entertaining even if I don't expect it to be true (to be clear, I'm approaching this as someone that mostly consumes his podcasts, which some chapters of his books have been directly converted to. I wouldn't want to spend the time reading this content that's not true, but I'm happy to listen to it while on a drive).
Whether he's causing more harm than good overall to the public is another question, but honestly that's a much bigger discussion with a lot of much bigger problems than him, so I'm not sure it's worth getting worked up over.
Stories/memes/narratives are the most easy/potent form of mental ingestion - so much so that I think humans cannot ingest facts or ideas at all, only stories. And this puts a collective responsibility on all of us to be very careful about the stories we create.
And that could be for a number of reasons. Of course, sometimes the results are just wrong, due to statistical flukes, or too creative data cleaning and analysis. Often the results might just be much more limited than what the original study claims: Maybe the results of a psychological study is valid for MIT students in the beginning of the semester, before lunch, but not for Yale students in the early afternoon. In this case the only mistake would have been to assume the results were universal.
It is amazing how many smart people have bad intuition on science, misunderstand the null hypothesis, etc. So much for the viability of a scientific thinking populace afaict: it seems not possible to pull off.
This tendency seems to be capitalized on fairly heavily in political media by finding some poorly supported assertion of the other side to criticize, which causes people to assume the opposite is true.
But such a small fraction of possible propositions are true that it is unlikely to be worthwhile to waste much time on propositions with no evidence.
I'll have you know you just nearly nerd sniped a mathematician ;-)
I suspect I may not be the first person to entertain this question, perhaps there is some literature on the matter.
Like, it was in all my cog psych textbooks more than twenty years ago, with cites back in the 80s (which weren't him).
This is my favourite paper of theirs: http://stats.org.uk/statistical-inference/TverskyKahneman197...
I got into a bunch of trouble with some reviewers of my thesis for referencing this repeatedly.
I don't know if there's a better text on dual-process theory out there (perhaps by the original authors), but regardless of who originated it, I think it's something worth learning about for everyone (and if you don't have a better source then Thinking Fast and Slow is a very good one).
https://journals.sagepub.com/doi/full/10.1177/17456916124606...
(that's a link to a defense of dual process theories, but it makes clear there's increasing criticism of them)
https://scottbarrykaufman.com/wp-content/uploads/2014/04/dua...
There's review paper coming from a more critical perspective in Psych Bulletin or Psych Review I was looking for, but I couldn't find it atm.
Irony is, Kahneman had himself written a paper warning about generalizing from studies with small sample sizes:
"Suppose you have run an experiment on 20 subjects, and have obtained a significant re- sult which confirms your theory (z = 2.23, p < .05, two-tailed). You now have cause to run an additional group of 10 subjects. What do you think the probability is that the results will be significant, by a one-tailed test, separately for this group?"
"Apparently, most psychologists have an exaggerated belief in the likelihood of successfully replicating an obtained finding. The sources of such beliefs, and their consequences for the conduct of scientific inquiry, are what this paper is about."
Then 40 years later, he fell into the same trap. He became one of the "most psychologists".
http://stats.org.uk/statistical-inference/TverskyKahneman197...
I imagine that in 30 years, it will become clear that individual humans display enormous diversity, their diversity increasing as societal norms relax, and their behavior changing as the culture around them change. As such, replication is hopeless and trying to turn "psychology" into a science was a futile endeavor.
That is not to say that psychology cannot be helpful, just that we cannot infer rational conclusions or predictions from it the same way we can from hard sciences.
Self help books are enormously helpful, but they're definitely not science either.
I don't think we - people used to STEM - appreciate how difficult behavioral psychology is. In STEM, we are used to isolating experiments down, so there are as few variables as possible. And we are used to well-designed experiments being reproducible if everyone does what they are supposed to do right.
In the study of human behavior there are always countless uncontrollable variables, every human is a bit different and it is very difficult to discover something that would apply generally. Also, pretty much all of the research is done on western population of European descent.
This is why I take all behavioral claims with a large grain of salt, but I still have respect for the researchers doing their best in the field.
Unfortunately the reality is that the more interesting and quotable the result is, the less likely it is to replicate. So replication problems most strongly hit things that seem like they should have the greatest implications.
Kind of a "worst of all worlds" scenario.
Some believe the resulting garbage. And wind up with weird ideas.
Others use that garbage to justify throwing out everything that scientists say. And then double down on random conspiracy theories, denialism, and pseudoscience.
I wish there was a solution to this. But everyone is actually following their incentives here. :-(
Beyond this specific issue, are psychology experiments and issues time and culture sensitive? I think so [1]
[1] https://www.ssoar.info/ssoar/bitstream/handle/document/42104...
My impression is that the priming chapter is bunk, but the rest has generally held up. Is that no longer true?
Where psychology is massively failing to replicate is in trying to characterise healthy individuals. Typically the work of Kahneman.
But that's what interest people and sells, pop psychology.
I'm only tangentially following the whole autism/Asperger's/ADD/ADHD development, and I'm growing more and more convinced that all these categories are mostly arbitrary constructs grown out of random history and academia politics. Happy to be proved wrong here, though.
How do you define science? Could it be a science, according to you, or is there something fundamentally non-scientific about it?
Some fields are quite lucky that the universe is so elegantly organized, but for that isn't true for the overwhelming majority of fields with as many degrees of freedom as biological systems and anything more complex.
That doesn't mean we can't conduct experiments that reproduce.
Science is that which could be disproved.
It is a very small, and very important, part of human knowledge.
See Chalmers' What is This Thing Called Science? for an introduction to these kinds of topics, or Kuhn and Feyerabend's work for historical responses. (And the Duhem-Quine thesis for the "auxiliary hypothesis" response to falsifiability I hinted at with my example.)