Hacker News Reader

Comment by mbell | Hacker News Reader

I think it's situations like this that result in society having such a hard time with 'science'. The term has been so heavily co-opted by fields that just don't have sufficient rigger for the term to hold weight. Yet, on various topics, we have this publicized attack of "your a science denier!". At the end of the day, there are two 'types' of science, one where I can take the results and make accurate predictions, and one where I can't. The later just amounts to 'our best guess' where the accuracy is entirely unknown. If we want the general populous to 'trust science', we need to stop calling the later science. In short, if you can't repeat and predict, stop calling it science.

View on HN

Scientists usually try to distance themselves by saying those are soft science or even pseudo-science. This leads to the embarrassment of the demarcation problem[1] which is that no one can give a bright-line rule[2] to distinguish between the "real" science and pseudo-science. All of the demarcation criteria that have been proposed (such as Popper's falsifiability[3]) are inadequate in one way or another. In particular, they don't seem to capture the reasons a scientist would give about why a particular nutrition or social science paper is bad. The scientist would say things like, "Well, your sample size is small and not representative of anything except psych undergrads, you didn't control for age or gender, the participants and experimenters weren't properly blinded, you tested 15 hypothesis and only reported the p-value for the one that was under 0.05, and even that is wrong because you didn't apply Yate's continuity correction on your chi-squared test, AND NONE OF THAT EVEN MATTERS because the effect size you report is too small to be of practical consequence!" Nothing in there about the hypothesis not being testable; yet this is the kind of stuff that really separately the wheat from the chaff.

So we're left with a "No True Scotsman fallacy" where have to say that some science is "good" and some is "bad" and the only way to tell is to ask someone knowledgeable to evaluate each paper on a case by case basis. Not terrible useful to the layman.

And why do we want any kind of science to automatically get respect anyway? Good science is good because its already been subjected to an incredible degree of scrutiny. It will hold up to a little more. The real problem is disingenuous, bad faith arguments which are allowed to dominate the conversation. The real problem is to teach the general public to distinguish between sincere, good faith arguments and patent bullshit. This is much more difficult than it sounds because bullshit can easily conform to any merely superficial characteristics.

[1]: https://en.wikipedia.org/wiki/Demarcation_problem

[2]: https://en.wikipedia.org/wiki/Bright-line_rule

[3]: https://en.wikipedia.org/wiki/Falsifiability

I think a lot of what you are talking about is due to a huge amount of 'papers' being nothing but coloration tests. Again, I think terminology is of utmost importance when discussing public perception so I would rather not talk about this as 'science'. It's 'research'.

I don't see a "No True Scotsman fallacy" here because I think I define it quite clearly: Can you provide accurate predictions? I'll concede that there exists a bit of grey area in that question, but the answer is heavily bi-modal.

Why not have a general checklist with a minimum set of requirements for scientific papers that are relevant across all branches of science? The people putting their names on the paper would have to show that they followed everything or give reasons for skipping a step. The receiving journals would have their editors re-check the checklist. As part of reporting the results of the paper, the level of completeness of the checklist would also be in the report.

Yes, the checklist would not be all-encompassing or foolproof, and there would likely be revisions to the checklist, and maybe even domain-specific variants, but it would be an extra level caution that the media could report or choose to ignore at their will. Over time, the apparent level of scientific rigour would improve. No, it’s not bulletproof and there will be people who will try to meet the checklist and still present erroneous conclusions as reliable science, but it would be an improvement in the status quo for a layperson who is aware and values said checklist.

There are things like CONSORT[1] which kind of do this. Statisticians like Fisher[2] have a ton of good general advice of the design of experiments. (A plug for The Lady Tasting Tea[3] and the 7 Pillars of Statistical Wisdom[4] feels appropriate here.)

On the whole though, most of the things you should and should not do are so domain specific its very hard to give much useful advice at the level of "all science." Right now this seems to work because researchers are so eager to anticipate objections and and avoid unnecessary arguments during peer review they stick slavishly stick to the same methods used by seminal papers in their field, and this has the same effect as running down a checklist.

There probably is a case to be made for using an actual checklist, though[5].

[1]: http://www.consort-statement.org/

[2]: https://en.wikipedia.org/wiki/Design_of_experiments#Fisher's...

[3]: https://en.wikipedia.org/wiki/The_Lady_Tasting_Tea

[4]: https://www.goodreads.com/book/show/27311742-the-seven-pilla...

[5]: http://atulgawande.com/book/the-checklist-manifesto/

> which is that no one can give a bright-line rule[2] to distinguish between the "real" science and pseudo-science

"if you can't repeat and predict, stop calling it science" seems like a nice bright line.

Peer-review obviously isn't enough, I'd like to see peer-replicated studies become a thing.

What about astronomy, cosmology, archeology, paleontology, volcanology, evolutionary biology, cladistics, macroeconomics, etc., which do not allow us to "repeat and predict," as you say?

Some of these cases can be rescued by considering "retrodiction"[1][2] as valid substitute for prediction in the right circumstances, but not all.

I personally think the analysis of the Mott problem[3] points the way to the solution to some of these kinds of issues. That is, a prediction can take the form of a likelihood function which assigns high probabilities to certain combinations of events and low probabilities to others. Theories with low perplexity[4] can be considered correct even if they can't make predictions, and the study of such theories can be scientific. But as far as I know I am the only one who thinks so.

[1]: https://en.wikipedia.org/wiki/Retrodiction

[2]: https://afdave.wordpress.com/2007/09/04/sir-karl-popper-and-...

[3]: https://en.wikipedia.org/wiki/Mott_problem

[4]: https://en.wikipedia.org/wiki/Perplexity

> What about ... which do not allow us to "repeat and predict," as you say?

"if you can't repeat and predict, stop calling it science" seems like a nice bright line.

For what it's worth, there is science meeting the "repeat and predict" definition that can be done in each of the fields you list.

Astronomy / cosmology absolutely makes testable predictions.

Not that I necessarily agree, but the suggestion was to find a different word for these fields.

(Also, most of the fields you listed do allow for replication.)

even statisticians call social science - psychology especially - a pseudo science

For me personally, situations like this make me trust science that much more. If someone finds that a previously published paper contained errors, as any human work is likely to, and the response is to go back and correct those errors or publicize whatever was wrong with the research, well that's science working as intended. That's freaking awesome.

What I get really, really, really frustrated with is the attitude that some people have where retractions of stories or articles makes them trust the source less, rather than more. If a journal publishes a retraction, I have that much more faith in that journal, and that much more faith in the scientific process in general.

Similarly: news outlets retracting stories or issuing corrections makes me trust those outlets that much more, because it shows that they care about making a best effort to present truth, and they care more about their reputation as being a source of facts than they do about whatever short-term backlash there might be.

Any media outlet, journal, web site, or other publication that doesn't regularly issue retractions and corrections is not to be trusted. It'd be better if the erroneous information didn't get out there in the first place, but there's no scenario where everybody gets everything right the first time, all the time- it's going to happen.

You can really see the different perspectives highlighted from people posting to social media in response to things like changing estimates of the age of the universe. There's always a contingent of outraged morons screaming stuff like, "now they're saying it's 5 billion years older than they thought it was before, and people continue to trust them?!? All those scientists are such hypocrites for doubting my belief in anti-vax/Austrian economics/chemtrails/Noah's Ark/etc., how dare they!" They act like changing your mind based on new data, or admitting you were wrong about something is a sign that you shouldn't trust someone, whereas I would say the ability to constantly revise your beliefs is a fundamental requirement for trusting someone's judgement. The inability to do so is a reason to not trust anything someone says.

Well then, I'll create a journal which constantly publishes garbage and instantly retracts 99% of it. By your logic, the remaining 1% should be ultra, iron-clad trustworthy! :)

Scientists bear a public responsibility. Studies like these are used to justify public policy, opinion pieces, etc. If a civil engineer makes a mistake that impacts the public (say a malfunctioning building structure) that could have been prevented were it not for lack of controls, he or she would be investigated, and potentially be basically blacklisted in their field. If a doctor made a negligent mistake because he was unaware of how to use a tool and a patient injured, he would be held accountable.

In this case, social science analyses are used to answer questions of major public importance. Governments are constantly trying to reduce suicide rates, and make their populaces happier. Papers making claims that religion makes children selfish and unhappy are used to make public policy. In this case, if this paper were used as justification for legislation, we now know that the policies it would tend to suggest would be bad for the population. Someone, somewhere probably ought to be hold accountable in the same way as any other professional. I do hope that journals take appropriate precautions with this researcher in the future, and that the peer reviewers assigned to this case are duly sanctioned. This is a complete failure as professionals.

There are situations in which scientists can and do get things wrong through no fault of their own. For example, during the highly publicized EM drive tests a while back, an initial NASA report indicated that thrust was observed after careful evaluation. This is fine... they reported what they saw. However, after some additional engineering and measurement tuning and stronger sensors, the thrust was attributed to another source, so the claims were retracted. This is science. At every step the scientists demonstrated competence and professionalism. Nowhere did anyone say 'oops we forgot to use the sensor the right way that's why it didn't work, and in the meantime our paper was used to engineer other solutions'. There is a fundamental difference between being wrong and misrepresenting what you saw, whether through mistake or ignorance.

I've always thought this is what gives science a strong position. Science can always reverse itself and change positions (180 degrees if needed) to support the latest findings.

Politics, religion, etc. don't have the same advantage.

So here's the issue. There is a segment of the population that is skeptical and a segment of the population that is not. When the skeptical segment expresses distrust of a scientific paper and names specific grievances, oftentimes the unskeptical population will accuse them immediately of being 'anti-science'. This is a major problem, because when -- inevitably -- some problem in a paper is acknowledged by the wider 'science' community, the unskeptical 'pro-science' people who have framed the entire issue as being 'for' or 'against' science automatically make the less educated skeptical people all the more skeptical of science, until they actually become the caricature they were believed to be.

The solution to this of course is for everyone to remain more skeptical when it comes to science, and -- for the times when you do have reason to believe particular studies -- to respond in good faith to other skeptics, rather than to resort to name calling. If you are unable to defend why a certain piece of research should be believed, you should probably either accept that you are not educated enough to be able to comment and thus are also not doing 'science', or that the guy you disagree with may have a good point and the researcher in question bears the burden of proof.

On it's own, I think scientists making bad research and then retracting it would not cause people to distrust science. The 24-hour pop science news cycle combined with the massive rise of scientism as a religion (and the skeptics the equivalent of a medieval atheist) has.

> There is a segment of the population that is skeptical and a segment of the population that is not.

I think you missed the biggest segment: not paying attention. It's for these folks where terminology and clarity is important.

I'd divided the world into more than that. For me most of the world is superstitious (I realize that's my judgement). In my experience 95% of the people I meet believe in homeopathy, acupuncture, aroma therapy (not just that things smell nice but they they have strong medicinal benefits), spirits, ghosts, astrology, blood type personality, supreme beings, MSG causing headache, changes in weather causing colds, airconditioning making people sick, blowing fans causing sudden death, and a million other discredited things. Some of those people overlap with the type of skeptics you mention but most are not

That's not the problem here.

A statistical error was made, published, and then corrected.

That's possible in physics, chemistry, or biology just as well.

> That's possible in physics, chemistry, or biology just as well.

Yet it seems to happen far less often, or at the very least, in less publicly impacting ways. Part of this is the media, but part of it is the participants themselves. If you discover something new in physics for example, either the public doesn't care, or they don't really know, they just get a faster iPhone processor a couple years down the line when the predictions hold. The folks in your field are even skeptical at first glance, "do you have a 6 sigma result?". "Ok, well, lets talk then, but I still wanna see it reproduced". Psychology on the other hand, some person does a half assed 'study' and uses it to claim knowledge of some important aspect of humanity.

I'd really encourage looking at the Higgs discovery press conference as a perfect example. To my recollection, there was little to no mentioned of the Higgs, just cold hard facts, perhaps at the end there was a 'this is consistent with the Higgs'. Only months later were many of those involved even comfortable enough with their level of certainty to really say 'this is the Higgs'. They are searching out knowledge, and don't want to declare having found it unless they are certain. This is _good_, it's what we want science to be, the summation of our current, highly confident, view of the world. We don't want 'science' to encompass all untested and unproven hypothesis about the world.

The difficult point which I have to concede is that terminology is important. Certainly many folks in physics that maybe haven't hit on a predictive result would like to be recognized as scientists, and rightly so if they are on the path toward this endeavor. But the populous is simple, they want clearly defined words, they want 'science' to be known fact. Outside introducing new words, I don't know how to resolve this. If we don't define 'science' as denoting that which is rigorous, then we can't use "science denier" as a term, regardless of the topic.

> Yet it seems to happen far less often, or at the very least, in less publicly impacting ways.

Does it? Do you have any data to indicate that, or even a good way to define your terms?

Your argument seems to be purely emotional disdain.

> Yet it seems to happen far less often

Citation needed. And if so, that is a reason to draw a line where on one side is "not-science"? That is just absurd. Does a car A stops being a means of transport when it subjectively breaks down more often than car B?

> very least, in less publicly impacting ways

Excuse me?

* A literal century ago someone failed to translate a German study so know about every child in the western civilization gets a good dose of distrust in science when they get indoctrinated that the tongue has separated regions for taste which is ridiculously easy to refute for yourself in about 15 seconds.

* The coup of the cereal industry to fund some studies telling everyone that breakfast is the most important meal of the day still misguiding health guides today.

* Schrödinger telling the world how stupid it would be to assume quantum principles in the visible world, still happily recited with the complete opposite meaning by about 500 media entities per day.

* Scientific entities failing to have any impact on people about the dangers of X-rays until people got impotent from having their shoes measured via X-rays in the local shopping mall

To be clear, this is not intended as some sort of smear campaign to science itself. I want to illustrate that all science is vulnerable to even dumb mistakes and that this dumb social sciences ain't real meme is only slowing down much overdue conversation!

You're supporting my point here, all your examples did not make an accurate prediction. Thus, they are not known 'facts'.

Your definition of 'science' is 'the best we know', mine, and I think what is meaningful for public discourse is 'this is true'.

Sure, but what's your definition of "true"?

Derivable from first principles? General consensus? Observed once and seems to fit with the current model?

I don't really now how to make it more clear: Can I make accurate predictions from it? That's it.

"One idea is truer than another if it allows us to explain and understand more of our experience.

The idea that the sun and stars move around the Earth explained only why they move across the sky, but the idea that the Earth orbits the sun while rotating on its axis is more true, because it explains also why we have seasons. Strictly speaking, however, we will never know whether the Earth really revolves around the sun; another, even truer, theory could conceivably come along.

In support of his view, James pointed out that in practice all scientific theories are approximations. Rarely, if ever, does one theory explain all the facts of experience. Instead, one theory often does well with one set of phenomena while the other theory does well with another set.

A scientific theory that explains more is truer than one that explains less, and the truer theory is preferred. Kuhn might add that even a paradigm that explains no more phenomena than a rival but explains those phenomena better is preferred—as for example Copernicus’ heliocentric model of the solar system was preferred to Ptolemy’s geocentric model, because Copernicus’ model was simpler and more elegant that the cumbersome epicycles of Ptolemy’s model, even though at the time the two models fitted astronomical data about equally well. If scientists prefer theories that explain more phenomena and paradigms that make more sense of our experience more plausibly, then the progress of science no longer seems so unreasonable. It is the result of selection, the exercise of scientists’ preference for theories and paradigms that make better sense of our experience."

Taken from the book "Understanding Behaviorism" by William M. Baum.

I tend to agree. There is a plausible hypothesis to explain it too. Namely, that many social scientists don't particularly like doing statistics or calculations or that they are not naturally very good at it or that they are not very conscientious. None of these factors contribute to reliable analyses. People mostly go into social sciences when they are more interesting in people than in statistics or calculations.

A study friend of mine who studied an exact science ended up working in the social science department where he also was asked to teach some research methods class. Suffice it to say that even PhD students are not very good at it. Even at things where you would suppose they would be closer to their core competency, i.e., they were not that good at avoiding the pitfall of putting leading questions in a questionaire.

This is not simply a matter of social science being soft. This was caused by exactly 1 thing: researchers not releasing the code they write or use. Their code was bad, and it produced an incorrect result. There is probably TONS of this in research from the past 30 years, and most of it goes uncaught because, for some insanely unwise reason, the scientific community has tolerated researchers keeping their procedures, tools, and experimental processes secret... but only when software is involved.

Science isn't just a matter of what people believe. It often gets considered when governments create policy. Bad science can kill millions. I've run into many 'bugs' in the algorithm descriptions published in computer science and mathematics papers, I shudder to think what the code backing nontechnical research is like.

How does your criticism apply to this scenario? The conclusion of the study (if correct) would in fact provide some predictive power: one would be able to make some predictions about a random child's generosity given their religious upbringing. It should also be repeatable, assuming the methodology for gathering and analyzing the data was published.

They didn't confirm the model worked by testing the predictions. No successful predictions = not worthy of being considered as 'fact'.

The experiment may be repeatable, but that does not mean it's reproducible or even useful. Many of these "soft science" fields are plagued by

1) replication failures

2) scientific errors both accidental (coding errors) and intentional (p-hacking)

I think part of it has to be physics envy, where many disciplines are attempting to adopt either the statistical techniques used by physicists or at least similar language and ideology in order to conduct "experiments" in fields as varied as sociology and psychology.

This seems like cargo cult science. Who says that, say, measuring the tendency of churchgoers to donate to a cause will reveal a truth similar to the mass of an electron -- something that can be accurately measured once and then you know a good measurement will return the same result? It's not at all clear that any real insight is gained of either philanthropy or church attendance from such an "experiment", nor is it clear to me that these fields contain statements with truth values at all, at least truth values as would be expected by mathematicians, physicists, chemists, etc.

But by adopting the methodology of science to fields which may not have yield scientific results to yield, a lot of people are creating a body of fake knowledge, or the appearance of knowledge.

Have you never tried to use a tool from a cs paper and found that their code didn't even build or barely worked? This isn't a question of rigorous and non-rigorous fields. This is a challenge with artifact evaluation in all fields.

I'd encourage you to read some of these 'papers', there is often nothing to evaluate, nothing to make predictions on. Usually, just a survey and a statistical correlation test.

I think the bigger issue is intellectually lazy readers and media which don't really care to understand nuances or spend any effort reviewing and fact checking things.

You can't control labels, no one can control what is labeled science or not, we just need to educate people, demand more of our media, and breed a culture that take more pride in truth, accuracy, precision, and intellect.

> I think the bigger issue is intellectually lazy readers and media which don't really care to understand nuances or spend any effort reviewing and fact checking things.

Sure, I don't disagree, but do you have any path toward this 'enlightenment'? I mean, the average college attending student has a literacy level of about 7th grade. How would you propose we move from there to a point where even the average, much less most, voting age adults can not only digest a paper but also be capable of poking holes in its reasoning?

> You can't control labels

Of course you can, it's what politics and marketing are all about.

I partially agree. Perhaps you comment is related to Brieman's Two Cultures? Paper: https://projecteuclid.org/euclid.ss/1009213726

I think you're right that they are two different things, but I'm not sure that later isn't science.

Perhaps the real problem is that reality does not punish being wrong and reward being right enough.

Another thing that needs to stop is the over use of the word "theory" where "hypothesis" is more accurate. But everyone want to be a scientist with big ideas.

I think you're looking for the word rigor

Computer Science

>rigger

uh... rigour

Rigor! https://www.thefreedictionary.com/rigor

that's what i thought, google only showed the medical definition at first, and i trusted it

Either is good. https://en.wikipedia.org/wiki/Rigour

Rigour is the correct spelling in the UK.

Reply on news.ycombinator.com