Hacker News Reader

Comment by dekhn | Hacker News Reader

Many details of this particular experiment made me greatly reduce my confidence and interest in social science. I was trained up in quantitative biology- and when I look at studies like this, I see a long list of "things that could go wrong, leading the investigator to falsely conclude their hypothesis is true". But in this case, I think the investigator actually didn't care enough about doing high quality research- they simply started with a moral belief/value judgement and ran an experiment and chose to interpret the results to support their "hypothesis". And the nature of social science is such that it's really hard to truly run an "honest experiment".

View on HN

You can read over a hundred years of extensive, exhaustive criticism of most social "sciences" for exactly that reason, and many of us grew up with a categorical understanding of a difference between "hard" material sciences like physics and chemistry and "soft" sciences like social sciences and many subdomains of biology and medicine.

But that distinction has largely fallen out of the zeitgeist and many people now just take anything ever published in a "scientific journal" as sound.

It represents a huge regression in scientific literacy among the public and sets us up for people becoming increasingly skeptical of "hard science" conclusions because so much of what they've incorrectly come to accept as science never really was.

True but there has been a movement towards replicating these high profile findings in the soft sciences. Hopefully that will gain more traction as a lot of the "newsworthy" studies are forced to get retracted after failing to replicate.

> people becoming increasingly skeptical of "hard science" conclusions

A big problem is that "hard science" conclusions often only apply to very specific circumstances, but scientists and the general public then extrapolate to more generic situations. The consequence is that a lot of things that are supposedly based on "hard science" aren't really proven at all, it's just someone making educated guesses.

PSA: There's also a severe replication crisis in the hard sciences. The high horse is not well warranted.

Completely false equivalence. The entire foundations of modern Psychology are wobbling. In order for the same to be true for the hard sciences we would need to be failing to replicate experiments which hinge on germ theory, atomic theory, the standard model, etc.

Nothing like that is happening. This false equivalence originates from several types of people:

1. Journalists that want/need to foment the largest possible catastrophe.

2. Political pundits which want/need to discredit some field.

3. Social scientists playing defense.

What exactly do you think are the foundations of modern psychology? Serious question.

There are tons of non-replicable findings way, way further down the stack than psychology, and those tend to have a lot more relying on them than psychology/sociology studies. If you're upset about scientific validity, consider directing your ire to where problems are more likely to actually hurt people -- the "hard sciences."

Nice ad hominem but I'm none of those things. I work in clinical trials, one of the few areas where we actually do have to know things, and a very good empirical demonstration of exactly how incredibly difficult that is.

I'm curious to hear your perspective on the validity of psychology / psychiatry / sociology as someone adjacent to the field.

I am a hard science maths / data science guy, but unlike a lot of my peers I have a great interest in softer reasoning (philosophy, ethics, political science etc). But I am constantly disappointed by how tainted by ideology psychology and psychiatry feel (and economics, but this is a different discussion).

Do you think that psychology and psychiatry are held to the same rigour as harder sciences and should be considered as valid?

All fields of inquiry are tainted by ideology. Read the history of literally any scientific field ever. The entire system is designed to accept this as fact, because science is done by humans, and to still arrive at the truth nonetheless.

If we take those two observations: a) science is done by humans and b) humans have motivations, obviously the way to arrive at truth is to allow for most things to be wrong most of the time. This is the process by which we've learned every single thing about the universe.

I don't know what you mean by "held to the same rigor." I don't think any psychologist on the planet would tell you we understand psychology as well as we understand basic chemical reactions.

> The entire foundations of modern Psychology are wobbling.

What do you mean by this? the field of psychology is perfectly capable of policing itself, and it's rejected much of the conclusions of its historical predecessors.

> In order for the same to be true for the hard sciences we would need to be failing to replicate experiments which hinge on germ theory, atomic theory, the standard model, etc.

"hard sciences" also fail to produce results relevant to most people. Sure, they can maybe make better batteries, but how can they explain how dysfunctional society is?

EDIT: We can also directly blame the poor communicational skills of "hard sciences" for diet culture. "hard sciences" have failed in their efforts to produce a population that can reason about nutrition in an evidence-backed manner, and this can be traced directly to how scientists choose to present their data.

You don’t need science to explain how dysfunctional society is. The problem with “soft sciences” is that they can’t produce provably correct information. Incorrect information is worse than no information.

> You don’t need science to explain how dysfunctional society is.

You absolutely do lol, or you're just straight wrong. Take your pick.

> The problem with “soft sciences” is that they can’t produce provably correct information.

Neither can hard sciences. Proofs are incompatible with empiricism. Abductive proofs (which is what the scientific process offers) are necessarily bounded by limited certainty. All you can do is progressively improve certainty approaching 100%, but reaching it is necessarily impossible. This is just basic Hume. You can never be 100% sure the sun will rise tomorrow or that the fundamental laws of physics won't arbitrarily change.

But, the same applies to soft sciences. We can and do increase our certainty continually. This is absolutely worthwhile and is probably far more valuable to humanity than merely modelling physical phenomena.

> You absolutely do lol, or you’re just straight wrong.

This is false, an obvious counter-example is an explanation that happens to be coincidentally correct (without being derived from any science)

This seems like an attack on the notion of objective truth which while interesting in the abstract, is not particularly interesting to me personally.

> an obvious counter-example is an explanation that happens to be coincidentally correct

How can you establish "correctness" without something like the scientific method? How do you even bind loosely-defined english to verifiable claims to real-life referents without agreeing with others on terms? No, you're chosen simple, comforting delusion over anything resembling objective truth.

> This seems like an attack on the notion of objective truth

Yes, truth is an apriori concept; objective truth is a silly delusion. Coherence is generally a much stronger concept anyway.

Correctness (of a given assertion) doesn’t need to be established to exist

> "hard sciences" also fail to produce results relevant to most people.

I don't see how you can say this, would you prefer to live 200 years ago before hard sciences had started changing peoples lives? Almost every convenience you see around you exists thanks to hard sciences.

That's an awful lot to put on hard science rather than, say, industrialization—the result of soft sciences just as much or even more so than hard sciences. Plus if you insist on only evaluating material concerns we still need to grapple with soft sciences to figure out why society is so horrible at distributing material goods and services in a rational manner.

My point was not to reject hard sciences so much as to emphasize you can't easily extract the consequences of them in isolation. It's nearly futile to even try. My apologies for poorly articulating this.

To be fair all of that was up in question a century ago in the hard sciences. People used to believe in the plum pudding model, or doubt what component in the cell contained genetic information. The only thing that changed that was incremental experimental evidence. Social sciences are evidently going through a similar transition but we shouldn’t use that to discredit the field alone, work is still able to stand for itself if it is done well.

The level of ignorance of this basic, basic historical fact is just completely astounding on this website sometimes.

It was called natural philosophy for Christ's sake.

Modern medicine's forefather was heroic medicine, based on modulating the 4 humors (blood, phlegm, yellow bile, black bile).

We didn't zap into existence with the hard sciences being hard. We made them that way through centuries of intellectual effort, almost all of which turned out to be wrong!

True, but I remember reading the replication failure rates were twice as much in the social science than in medicine.

We constantly see small medical studies (<100 participants) posted here on HN that produce exciting results, which then disappear from view and/or fall apart when replicated with larger cohorts.

How is this different from science working?

When it happens in the hard sciences its science working, when it happens in the soft sciences its a replication crisis.

Haha that seems to be the case!

If that didn't happen, it would mean that math was broken. That's how statistics is supposed to work.

Depends what you count as hard science. The replication rate in high energy particle physics is near 100%? When the LHC started up they were able to measure nearly all of the particle resonances found in the 20th century. It's not like they suddenly disproved the existence of electrons or something.

Sure, further down the stack of chaos that is the universe (physics -> chemistry -> biology -> psychology -> sociology), it's much easier to conduct controlled experiments.

That doesn't mean the people engaged in research at the bottom of the stack are good and the people at the top of the stack are bad. Nor does it mean we shouldn't be trying our best to understand things near the top of the stack.

> it's much easier to conduct controlled experiments.

Very true. But this means more statistics and controls are necessary to get solid result from a social science experiment then a particle physics experiment, no? Clearly, this is practically impossible, but there you go.

> Clearly, this is practically impossible

No it's not? You put more money into the studies and you can do bigger, better versions of them.

A major obstacle to putting more money into studies: people jerking themselves off about how soft sciences are a joke and hard sciences are Super Serious Business.

But why do those fields deserve more money, when at least a large part of the problem is cultural.

One example is the famous reluctance to publish negative results in psychology. Nearly all published results in (collider) particle physics are negative.

If senior faculty prefer to only hire people with a string of published postive findings, you are literally encouraging p-hacking. Again, they are not "bad" people, it is just that the system the senior people have setup in that field is not conducive to doing good science.

> But why do those fields deserve more money

Because it'd be good to understand what makes people happy, for example. Or what enables relationships to thrive. Or when different forms of government are suitable or unsuitable to solve a set of problems, etc.

Sorry to break it to the hard-sciencers, but the vast majority of opportunities left in the western world to improve people's lives is not particle accelerators, it's answering questions like: "what actually helps people feel satisfied in life, loved in their relationships, and belonging in their community?"

> At least a large part of the problem is cultural

Is it? Why so?

Negative results aren't published in almost any field, and that's actually a good on ramp to the discussion we should be having, which is about the broken incentives of science and scientific publishing specifically. The broken incentive model isn't special to softer sciences and it has far more dire consequences in other domains.

You can't possibly think that soft sciences are the only ones hiring people with a string of positive results... right?

I agree that studying psychology better could be beneficial. Is it possible? Or more to the point, is it merely a matter of money, as you said?

I said a large part of the problem is cultural, I did not say that psychology is the only field with cultural problems. I'm not sure how you got that idea.

No no, what you said is that it's "clearly practically impossible" to have more statistics, more controls, etc. to get higher powered studies of high-chaos questions like the ones asked in the soft sciences.

I said, to that point: no it's not. You just do bigger, longer term, more complete studies. The limiting factor on this -- right now -- is typically money. Perhaps you can pour infinite money and problems with e.g. recruitment or monitoring will still prevent us from getting to statistical power, but maybe not.

That is not the only problem social sciences face, but most of the problems they face are not exclusive to social sciences whatsoever, which then prompts the obvious question of why they get so much flak.

I never said anything about practicality, you are confusing my replies with someone else's. I said this

"But why do those fields deserve more money, when at least a large part of the problem is cultural."

More money won't fix the cultural problems with the field. Maybe the lack of a universal quantitative framework makes issues that are slightly bad in some fields much worse in the social sciences. I don't know. But it's naive to just say other fields have the same cultural problems so we can ignore it for MY field.

There's a famous anecdote where Dyson came to Fermi with a theory for weak decay (I think). There were a handful of datapoints that matched experimental results.

However, Fermi threw out Dyson's theory, despite the empirical agreement, because it had 5 free parameters ("with 3 parameters I can fit an elephant, and with 4, I can make it waggle its trunk is the quote" IIRC). This is difficult, but essential criticism all fields need.

AND all this is apart from the fundamental question of whether important generic scientific truths can really be gleaned from the social sciences.

> Sorry to break it to the hard-sciencers

Believe me, you aren't "breaking" anything to anyone. If you could solve the secret of happiness (your example), no amount of money would be too small.

The issue isn't whether social science would be good to figure out. Definitely it would, to the extent there is actually a "thing" to figure out, which may be true and may not; i.e., "what makes people happy" may be so contingent and/or so ineluctably open to interpretation that it makes no sense as a rigorizable concept. (There is nothing wrong with unrigorous concepts, btw, these have been fruitfully explored by the poets and philosophers and therapists.)

Ok, so even granting that there is a stable, rigorizable "truth" for the social sciences to discover, the issue is whether the methods and analyses as they have been practiced are effective or even could be tweaked to be effective. Clearly, they aren't. And not just a few bad apple studies, but seemingly the whole darn lot.

> Clearly, they aren’t

Arguing is easy when you just assert your conclusion eh?

The question of "do those fields deserve resources" is answered as follows: are there interesting questions in those fields that we should ask and have answered (well)? I think the parent poster is saying: yes, there are.

This question is orthogonal to the question of whether the organizations currently conducting research in those areas are well-organized. You could fund them well and also demand re-organization as a condition. You could even find other scientists to do this work. But if you don't think the work is important, none of this matters.

That is fair. Is what you suggest possible though?

As my sister who is studying one of the soft sciences put it to me when i pointed out the lack of rigor compared to the hard. "sure we could make psychology a hard science but pesky ethics boards wont approve me raising batches of several hundred human clones in controlled environment for each test"

I mean, a psych experiment will never have an N comparable to a particle physics experiment or be able to reach the 5-sigma threshold for discovery that now prevails in physics. On the other hand, the object of study for psych is intrinsically interesting since we are people and if something reliable can be gleaned then it's certainly worth money. My concern is that "bigger, better" (as you say) would have to include millions of people across cultures and times, tracking longitudinally, with randomization and controls. (Again, more complexity requires more statistics, not less.). Is this practical? Maybe ...

I didn't claim that it means they are "bad" whatever that means.

I am saying that we should take those claims less seriously, especially if the results from that domain don't replicate, as in the case of psychology and other social sciences.

Maybe there is little we can conclusively say about those domains.

Yes we're in agreement on this.

There was little we could conclusively say about any domain until long term, concerted effort was made to understand each of them.

Anecdotally, I have a friend that's doing bio-chem doctorate and she has said that replication rate on that is abysmal for biology, chemistry and consequently bio-chem.

I'm sure some areas of physics have near 100% and some simply don't.

Did the public ever read published papers until recently? I can’t imagine most people having access to any publications until the Internet and late 90s at minimum.

My local library did not exactly have access to journals either.

I don’t think scientific literacy has ever been high. Society relied on other publications and the government to interpret the information for us. For better or worse.

University libraries still get hardcopies of monthly journals. Maybe back in the day a downtown central branch might have as well.

> You can read over a hundred years of extensive, exhaustive criticism of most social "sciences" for exactly that reason

There's nothing essentially non-scientific about the fields; it's just harder to control variables. The entire "hard science" vs "soft science" beef is a little silly when "hard science" isn't equipped to reason about most human concepts. Try not to chuck the baby out with the bathwater. I'd prefer to stop differentiating between the two ends of the spectrum as if they're inherently different.

I also find that people who poo-poo "soft sciences" still have strong beliefs about humans, society, etc, they just don't even bother trying to ground them in evidence.

The non-scientific aspect is that conclusions are drawn even when the variables haven't been controlled (because as you say most of the time it's basically impossible).

I imagine it must be really frustrating for people actually trying to learn facts when the famous results are dominated by nonsense like this, power poses, hungry judges, etc.

It's not just in research either. E.g. in the UK they did a regression of missed school vs exam results, noticed that they were correlated and now it's a criminal offence to take your children out of school for holidays, even for a single day.

> The non-scientific aspect is that conclusions are drawn even when the variables haven't been controlled

Yeah, it is so strange when people say we should listen to the scientists, we shouldn't since scientists make all these much stronger claims than the data supports.

Instead we should listen to the data, see what it says, rather than the scientists who did the studies. And it is really important to note that the data doesn't necessarily say what the study author says it does.

Edit: Above is for social sciences, since they lack robust results. In the hard sciences were we have a lot more robust results it makes more sense to listen to those who know those results, that doesn't have to be scientists though could be experts like engineers etc.

There's no rubrik for grading scientific correctness. Just because controlling variables is more difficult doesn't mean we should abandon the scientific method and just talk out of our asses.

The reality is social sciences are inherently more complex because they are higher-level problem spaces. Biology in merely a stepping stone to psychology, like assembly to modern Java. Assembly is simple, understanding how the JVM works is hard.

Psychology intertwines just about every field you can imagine. Anthropology, neurology, biology, chemistry, you name it. They're fundamentally working on a level much, much higher than the hard sciences. So the sheer amount of variables is absurd, but also how they work together.

Yes, the devil always seems to be in the details in psychology experiments. Were the experimenters giving subtle cues to the child, and was this simply a test at how deftly the child picked up these cues? What was the exact wording of the “deal” offered to the child and would a wording change alter the results? Was the experiment conducted at a time of food scarcity or abundance? What were the prevailing cultural norms of how a child “ought” to behave? Would the results change if average child age was 6 months older or younger when experiment was conducted? What was in the drinking water and the air and the paint at the testing site? (With strong claims in the literature that all these are correlated with measures such as average population IQ!)

In the face of all these potential confounders, more statistics and controls seem necessary than, say, in a physics collider experiment on electrons (each electron possessing exactly two characteristics, location and spin, and all such electrons behaving identically regardless of location or time). Yet, even in this setting of simplicity and reproducibility, physicists have still found it necessary to establish a stringent, five-sigma threshold for discovery — 3 sigma anomalies come and go. Such a stringent threshold is unthinkable in psychology due to practical considerations. Ergo, it’s hard for to see how psychology can become a reliable empirical science.

I'm not really disagreeing, but the 5-sigma rule is there because the hypothesis is not formulated before you run the experiment.

If you make the hypothesis first, 3-sigma is quite enough. Many physics experiments do exactly that, but famous high-energy ones don't.

(That said, not having an hypothesis beforehand was very common in psychology before the 21st century.)

This is one of those rules of thumb that don’t make any sense to me as someone who works with data in the field. You conduct the same exact experiment with the same conditions and same data and get the same result. But whether you speculate on a hypothesis before or after suddenly changes the significance threshold with no actual change in the underlying data or method? Did you cast some ancient spell when you came up with the hypothesis or something?

> with no actual change in the underlying data or method

Proving a known hypothesis or deciding what you want to know after the fact are completely different methods.

> Did you cast some ancient spell when you came up with the hypothesis or something?

"I'll know it when I see it" is an incredibly vague way of doing science that requires extra rigor somewhere else to compensate.

Or, in a maybe better explanation, testing for multiple hypothesis is subject to this kind of failure:

https://xkcd.com/882/

So you need more data confirming your theory.

But if you state your single hypothesis beforehand, you are at the situation of the top square, with a high-confidence result.

I have read from BOTH psychologists and sociologists criticizing this concrete experiment for years. It was popular among internet crowd of fancying themselves as "cool nerd kids" who play experts on everything. But if you read more boring write ups by actual scientists, they complained for years.

Hell, even the "Dunning Kruger effect" is a misapplication of statistics.

The effect shows up even with randomly generated samples. Because there are floors and ceilings to the data. If you're low, you can only guess so much further down, so you're likely to overestimate your ability. If you're high, you can only guess so much further up, so you're likely to underestimate your ability.

> Many details of this particular experiment made me greatly reduce my confidence and interest in social science.

There is a reason why many scientists diplomatically classify social "science" as a soft science. Less diplomatically minded scientists like Feynmann call it pseudoscience.

I mean let’s get real. This particular experiment is pretty irrelevant to how you feel.

There’s nothing that could be done in social science that you wouldn’t be skeptical of and want to dislike.

Reply on news.ycombinator.com