Cargo Cult Science (1974) | Hacker News Reader

Cargo Cult Science (1974)

calteches.library.caltech.edu

142 pointsfipar33 comments

> there have been many experiments running rats through all kinds of mazes, and so on—with little clear result. But in 1937 a man named Young did a very interesting one.

Has anyone found this paper? I searched a few years ago and couldn't find anything.

Me neither, and for this reason I tend to use this paper to see who pays attention. Best case: someone tracks down the missing paper(s).

I suspect it may be a joke Feynman played on us to prove a point. :)

Good catch.

As for "good catch": I think it took me at least 6 months from I first read the paper before I actually tried to find the research in question. Not finding even a single credible reference was a humiliating and humbling experience. I've looked for it several times over the year since and stille come up empty-handed.

I was giving a talk where I was going to refer to Feynman's speech and I felt I needed to fact-check it.

(I ended up using the speech in the talk I was giving, but I did give a hint at the end that perhaps this speech might possibly contain a joke at our expense, or something along those lines. So as to not look like a total idiot if someone else did a better job of checking the speech faster than I had done :-))

So kudos to you @killjoywashere :-)

Paul Thomas Young

obit: https://www.jstor.org/stable/1421573

rat paper still missing

Here's some stuff from him:

https://catalog.hathitrust.org/Search/Home?lookfor=%22Young,...

The 1945 one sounds closest: https://catalog.hathitrust.org/Record/012277991?type%5B%5D=a...

Out of all the experiments in the soft sciences that captured society’s imagination, one of the most egregious is “Power Posing”.[0] The idea is that you stand like a superhero in front of a mirror and all of these measurable chemical changes happen that improve your abilities.

The professor that discovered this phenomena had a famous TED talk, a bestselling book, switched her faculty position to Harvard, and basically rode that to success in her professional and public life.[1]

All of this came crashing down when no one could replicate her experiments. It’s absolutely incredible that things got this far in science, or at least the soft sciences. Shameful

[0] https://en.m.wikipedia.org/wiki/Power_posing

[1] https://en.m.wikipedia.org/wiki/Amy_Cuddy

There are more subtle variations on this that still affect the harder sciences.

A topic closer to home for most of us might be benchmarking. It is terribly difficult to construct an honest A/B comparison between two different algorithms, especially when the person doing the testing designed B themselves.

It's usually easy to get benchmark results that are reproducible within 10% or so, though often people don't bother. With careful work you may be able to get benchmarks reproducible to 9 significant figures. See SUPERCOP for an example.

Now, your reproducible results may or may not be fair, but that's an issue further down the line.

I dislike this usage of "soft" sciences. I think it helps to be specific about the field, the journal, or the community.

People use the word "soft" to describe any kind of scientific research that makes them uncomfortable, no matter the legitimacy. The social sciences are soft, psychology is soft, economics is soft, etc. It's a moving goal-post, when there are social scientists and psychologists who do very "hard" scientific research (and vice-versa).

Mathematics is "hard" because there is a provably right and wrong answer. Moving away from mathematics, subjects become much more malleable and open to interpretation. For example the fuzziness of psychology allows something like the "Power Pose" to exist, while in math there would be no "Power Matrix" or some such.

You tell that to the geometers.

A lot of applied math -- especially modeling -- is pretty freaking fuzzy.

I use "soft" to mean that the measurements are more subjective, contextual, or open to interpetation.

Chemical changes are "hard", but "power pose" and "improvements to your abilities" may be "soft".

Soft sciences are characterized by an inability to conduct controlled experiments. It's impossible to quantify all the independent variables so researchers guess or make assumptions.

They're called soft because the standards are very low when compared to the harder sciences, and it's basically impossible to do perfect controls or replications.

I wouldn't take this too seriously. Theoretically mathematics is also a "soft science" or no science at all depending on the science theoretical background of the person you talk to (science is only experiments with data, empirical stuff...you know, following the scientific method).

I have since developed a very soft criterion for what I call science. Roughly...stuff that advances our understanding of the world and is conducted in a rigorous way.

If measurable chemical changes fall under soft sciences, is there any hard science at all? By this definition, chemistry, physics, and biology are all soft sciences.

Yes, but how do you measure posturing? And if you can't (in an objective way) then the claimed cause/correlation doesn't hold up.

Things don't always work out this way though. If you publish a scientific paper that reports an interesting result, it can permanently help your career even if it is later proved to be wrong. It just needs to be believed for about 10 years or so - you'll extract most of the career advancement possible from the paper during that time. When the paper is debunked, as long as there is no evidence of fraud, you'll still have tenure, you'll still be the PI on projects you have started after the paper, you'll still have your network of collaborators who won't shun you if it looks like you made an honest mistake years ago.

I've seen the above happen, and it can be caused by a perverse fact one of my mentors told me in grad school: "Bad data almost always looks interesting." Cold fusion started out as bad data, and looked very interesting. Those faster-than-light neutrinos that were in the news several years ago were bad data. When I was a kid in the 1960s, there were numerous reports of exoplanet detections - all were bad data. Etc.

Somewhat interestingly, the rat-studying scientist Feynman refers to in the linked piece discussed body postures in animals to some degree in [1] back in the 1940s. No mention of Power Posing directly but it's just a curious coincidence. Maybe it means something! (Just kidding)

[1] https://babel.hathitrust.org/cgi/pt?id=mdp.39015026431919&vi...

Studying Statistics has ruined every "psychology" article out there.

>It’s absolutely incredible that things got this far in science, or at least the soft sciences.

In 1996, a physics professor submitted a nonsense paper saying that reality doesn't exist and gravity is a social construct. The reviewers didn't understand the paper, but published it anyway since it had all the right political buzzwords.

https://en.wikipedia.org/wiki/Sokal_affair

no one could replicate her experiments

... basically rode that to success in her professional and public life

Looks like the experiment has been successfully implemented in real life.

what about the facial feedback hypothesis?

https://en.wikipedia.org/wiki/Facial_feedback_hypothesis

Basically, the act of physically smiling may influence the emotion (usually it's the other way around)

Optimising software has frequently been very cargo cult-ish. No measurements at all or measurements that tell you nothing have been pretty common. It seems it's more common to take it seriously now, also tools are better and techniques get written up on blogs and shared, some of the good ones catch on, some of the bad ones eventually fall over. "If experiment doesn't match theory the theory is wrong." Feynman has been the source of a good bit of wisdom in my career and life. Re-post and re-discuss often. If you haven't, "Surely you're joking..." It's worth your time and will also entertain. I should read it again, it's been a while.

Many previous submissions; the ones with comments:

2016: https://news.ycombinator.com/item?id=13290107

https://news.ycombinator.com/item?id=11669004

2013: https://news.ycombinator.com/item?id=6543791

2010: https://news.ycombinator.com/item?id=1629571

2009: https://news.ycombinator.com/item?id=993150

https://news.ycombinator.com/item?id=723140

Experiments that attempt to create AGI singularities by just making huge neural networks strike me as cargo cult science.

There are people attempting to develop superhuman artificial general intelligence by making a single large neural network?

I've never seen that before but I have seen businesses:

* Use hadoop/latest distributed computing with data that fits on one machine

* Use way higher variance models than necessary for the problem at hand

And in general just cargo cult what the big players are doing when it's unnecessarily complex for the size of the business.

s/businesses/engineers looking for resume items for FAANG applications/ and it's completely rational.

"We obviously have made no progress—lots of theory, but no progress—in decreasing the amount of crime by the method that we use to handle criminals."

Not sure that's obvious. Crime has steadily fallen. Particular practices may not stand up to scrutiny, but I'm guessing a lot of it does kind of work.

In the US crime rose steadily from 1960 to 1990, and that's using the official statistics that don't even try to include things like the Iran–Contra scandal, the Gulf of Tonkin Resolution, CREEP, the Kent State murders, and the CIA protecting crack kingpins. Feynman was speaking in 1974, right in the middle of this period.

Reply on news.ycombinator.com