How a middle aged CS major debunked a classic positive psychology finding (2013)

The tile is in all caps "THE DUMBEST FUCKING THING I’VE EVER READ" and challenges someone to a cage fight at the end. The entire rest of the article could have been honeyed sweetness (which clearly it's not), and that could be enough to call it abrasive.

One person I showed it responded very favorably, commenting that the author clearly considered it "...important to get the basic points across violently vecause no one is listening to the non-violent arguments." i.e. this is abrasive, but abrasive to a purpose.

Did we read the same article? I'm confident that we can have actual communication that is both less obnoxious and not hamstrung by "bullshit etiquette."

Personally my biggest gripe personally is that it is represented as a model of total personality, but that's definitely not true. It's just just representing a larger "personality space" than most other constructs. At least the outcome isn't placing one into a discrete category.

That's pretty subjective and could be leveled against almost anything behavioral or psychological.

The thing about the Big Five is that it has surfaced in all sorts of contexts. You can maybe say maybe the 5 per se is not well justified (as opposed to 4 or 6 or 7, for example), but if you take enough ratings of a person, some variant of those 5 will probably work as reasonable summaries of the ratings, and they will account for a substantial chunk of their predictive variance. The thing too, is that if you take other types of variables, like clinical symptom ratings, or diagnoses, you start to see roughly similar types of attributes become prominent.

The Big Five is a descriptive model of how people perceive others. There's a lot of evidence for certain mechanistic processes being heavily involved in some (e.g., positive emotion in extraversion, negative emotion in neuroticism, behavioral control with conscientiousness, etc.) but I'm not sure the original idea behind the Big Five was mechanistic -- it was a hypothesis about major dimensions that could summarize social perceptual data. It's like classification in biology pre-DNA era. People have some ideas of how things go together, and find it useful for organizing descriptions and measuremnts.

It's like if you did unsupervised DL modeling of all the videos involving humans you can find on the web, and found that their classification could be accounted for by 5 major vectors, almost all the time, regardless of sampling. Wouldn't you want to know that?

Many other measures are very mechanistically well-justified but lack ecological validity, in the sense that they are very narrow predictively and not well outside of laboratory contexts. That's fine, there's a tension there between predictive bandwidth and depth, but if you want any kind of rating of a human being's behavior and experience, you enter at your own risk if you think you'll measure something that's radically different from the Big Five (or something subsumed by cognitive measures). Can you do it? Sure, but a lot of the time no (see: grit).

And yeah I think big five is useful in the above context, and also for things like this: https://arnoldkling.substack.com/p/keeping-up-with-the-fits-...

"Empirically, men and women to tend to differ on the trait that personality psychology calls agreeableness. More women show up as high in agreeableness than men.

[As an aside, I once wrote Nassim Taleb and the Disagreeables.

Nassim Taleb’s latest book heaps praise on the trait that personality psychologists call low agreeableness. . . I am pretty far out on the disagreeable end of the spectrum myself, but Taleb makes me look like a goody two-shoes.

Taleb came across the essay and tweeted this response:

There is this BS in this "disagreebleness" scale used by psychologists, unconditional of domain. Like most psych categorizations, BS. Many are socially gentle but intellectually rigorous & no-nonsense: others nasty in person but appear gentle in public . BS!

I rest my case.]"

e.g.

"Specifically, the mean of the 28 participants in the experimental condition, reported as 5.19, cannot be correct. Since all responses were integers between 1 and 7, the total of the response scores across all participants must also be an integer in the range 28–196. The two integers that give a result closest to the reported mean of 5.19 (which will typically have been subjected to rounding) are 145 and 146. However, 145 divided by 28 is 85714217.5, which conventional rounding returns as 5.18. Likewise, 146 divided by 28 is 42857121.5, which rounds to 5.21. That is, there is no combination of responses to the question that can give a mean of 5.19 when correctly rounded."

(that example is a fictional one but the same issue arises elsewhere)

In order to make definitive statements about said system, you need to start with the most basic behaviours and characteristics.

As you work your way up, you’ll mountains of data to capture every permutation of human behaviour and emotion.

We’re so, so far from any of that.

But to plug this fuzzy, professional knowledge into equations, call it science, and use it to make decisions in business and policy is at best naive and at worst fraud.

You are right that these largely don't exist in the Social sciences, but this is at least partially due to the much more complex subject matter. Still, both the social sciences as well as the natural sciences are trying to approximate reality through models built on experiments, and are thus fundamentally different from deductive reasoning in closed systems.

Not trying to mince words here, but engineering is applying validated knowledge to solve real world problems, sure, you can call it profession, but the fields generally associated with engineering are still in the business of generating knowledge. Medicine and Psychology heavily rely on the scientific method to validate abstract concepts, while engineering adjacent disciplines like Computer Science heavily really on maths and deductive reasoning to solve problems in the space of algorithms and computers.

Treating science as gospel is stupid. But without personal experience (and frequently even with) stating something as true is largely meaningless. Stating something is true, while providing data to back it up, as well as evidence that other subject experts have checked that data may not be perfect, but that doesn't put it on equal footing with a gut check.

Throwing engineering in there with psychology and medicine is not a very good grouping of disciplines. At least not if "engineering" means the kind of engineering that professionals in the field have legal liability for (like the engineering that goes into designing bridges and buildings).

Engineering does have "best practices", but those practices have to operate within limits that are very well understood based on underlying physical science that has been thoroughly tested and for which we have mathematical models with excellent predictive capability. The "best practices" in psychology and medicine have nothing like that kind of support.

> the process seems to work allright

Engineering, at least the kind that designs bridges and buildings, seems to work all right, yes. I'm not sure I would say the same for psychology and medicine.

It turns out that kids who are in stable situations with trustworthy adults around them, tend to trust adults more when they say “wait 5 minutes and I’ll give you more”, and that kids in bad situations don’t trust adults. Turns out being in a bad situation as a kid does have a long term effect on your life.

Nothing to do with the ability of people delay gratification.

Not every field is the same, they vary massively in size, level of external influence, and complexity in subject matter. Psychology suffers more than most. It's a large field, so competition to get noticed is likely high. There is money to be made, as discussed in the article, so it struggles with influences outside of pure scientific inquiry, and they have no chance of describing their theories in terms of pure component mathematics.

Contrast that with ocean and atmospheric sciences. It is a tiny and very insular field, so the rat race works very differently. This also protects them from outside influences, and a lot of the major underlying physical and biological processes involved can be modeled with mathematics that have been demonstrated to be predictive.

It's just easier for some fields to reliably produce higher quality work. I think this is a case where blanket skepticism is just lazy. Every field suffers some level of a reproducibility issue, but some fields are better able to self select for a variety of reasons.

Go to any CS conference and you'll see a whole bunch of papers doing micro benchmarks, saying that their method is X% faster than all the others, where X% is usually some very low percentage (and of course no errors bars are visible on any plots). You talk with the guy presenting the research and you find out that half the benchmarks were done on one machine, while the other half were done on another with different specs. Well, okay, now why can I trust the benchmarks? And how many other papers are doing this? And is your result even meaningful anymore? It's so depressing.

And don't even get me started on the AI/ML field. My god, talk about art over science.

Not every other field. We do have scientific theories that are reliable. But the reason they are reliable is that they are nailed down by extensive testing in controlled experiments that have confirmed their predictions to many decimal places. (I am speaking, of course, of General Relativity and the Standard Model of particle physics.) So we do know what actual reliable science looks like.

The problem, of course, is that there is not a lot of other science that actually looks like that, but it still gets called "science" anyway, and treated as though it is reliable even though it's not.

I think false rigor is worse than no rigor at all.

> In the linked article's example, if psychology were purely art, there would only be the authors' book, not a paper with actual math to debunk and a journal to interact with. I am probably overstating my point, because when a specious claim is backed up by scientific trappings people may be more likely to believe it, but I am increasingly less certain of that.

I think the primary reason why people are less likely to "trust" scientific results now than in the past is precisely because of some scientists generating "scientific trappings" with the appearance of rigor to justify a bullshit result. Most of the time, no one cares, and those scientists go on to their academic careers as usual. Of course, it's the follow-up careers, books, headlines that lead to the erosion of trust, not the fake math itself (that's just necessary to gain the approval of other scientists).

When that is attached to something that the average person can blatantly see is wrong, it erodes their confidence in the entire scientific process and establishment. They'll reject the entire concept of rigor before believing some of the things scientists claim to be true. How is a lay person supposed to separate the modeling that goes into e.g. climate change and "a 2.9 positivity ratio is the key to happiness"? To them they look pretty similar, a bunch of mathy looking garbage that tells them something that isn't intuitive.

Personally, I've been forced to sit through a lot of corporate management trainings that are full of citations of titillating psychology results that I know are bullshit. I just don't have the patience or motivation that Nick Brown has to properly debunk them.

Bitcoin had a few things going for it - the whitepaper is concise and well-explained, in perfect academic prose and typeset in LaTEX, which lends it an air of credibility. It came with an implementation, so hackers could experiment with it immediately, plus it sort of has implicit libertarian anti-government undertones, both of which encouraged follow-on blog posts.

Once the ball got rolling, another kind of heuristic took over - the money heuristic. People put money into the thing, which made other people go "oh look at how much money they put in, this is a real serious investment thing!" and put in their money, in a sick feedback loop that continues to this day...

Regardless of who Nakamoto actually was, it does appear that Finney and Szabo played a similar role that Sokal did in this story; getting support from high-rep folks was essential to getting Bitcoin off the ground.

When the Higgs boson was declared "discovered," the particle was found to exist with a five-sigma certainty. This brief article does a great job of explaining what that means: https://www.r-bloggers.com/2012/07/the-higgs-boson-5-sigma-a...

Physicists got to that point (very roughly) by constructing complex mathematical models and building large machines to test those models.

In contrast, leading psychology models change with the seasons, and large-scale human experiments cannot be (ethically) conducted with the control as seen in physics. Because psychology experiments have substantial environmental variability, drawing robust conclusions in psychology is exceedingly difficult.

This fact should mean that very-high-certainty findings in psychology studies must be subjected to the highest scrutiny.

Social science data is notoriously noisy and erratic. Relationships are notoriously weak. If your numbers have a lot of precision, more often than not it's just meaningless noise not "real" precision in measurement.

Wake me up when social science can reproduce their studies and I'll consider it a science. Until then, I consider it akin to "humours".

Even harder to tally: How many of those deserve to be more tainted because they saw it for what it was but didn't speak up. (Yeah, I know, hard to do for a lowly assistant.)

Using an irrelevant bit because it will get more clicks seems to fall under the "click bait" domain, especially when he was a grad from CS, not a current major.

Personally I would be more interested to know that it was a newbie student to an academic field debunked a core tenet. "New APP Students Debunks Core APP Finding" is much more interesting to me than a coincidence of the former academics pursuits with mine. It implies the question, "If a brand new student to the field saw this error so quickly, why didn't anyone else?"

Or, I could have made the point that social sciences as a whole are woefully non-quantitative and the real lesson is: ignore just about anything you hear about psychology studies in the media, because most of them come from innumeracy and invalid experimental techniques.

Another great point: when somebody, say an "average Nick", finds that a paper does contain an invalidating flaw, it can be challenging to get the attention of the establishment. Sometimes that's because you're an average Nick with no cred in the field, sometimes it's that the journal doesn't want to admit that one of their board has been publishing total crap for the last 20 years in said journal.

Instead, I summed up all those points in a single comment which was in no way shallow, but rather, from decades of experience with our broken scientific system.