Hacker News Reader

Comment by cynicalpeace | Hacker News Reader

40-60% of scientific studies are not replicable, therefore fake as far as I'm concerned.

That's why "show me the study" is such a midwit line of reasoning.

We need to go back to thinking more with first principles, received wisdom, and personal experimentation.

There's a place for scientific studies, but it's gone so far that many won't take an obvious point without a study behind it.

There are many such examples that are quite obvious but are hard to prove: processed foods are bad, screen time is bad for kids, starship won't fall on a shark etc. But people love to poke holes in obvious truths, especially in well educated spaces like HN.

View on HN

I think describing it as 'fake' is rather antagonistic. You could just as easily say 'poor quality' or 'unjustified'. I would say that more accurately describes the situation.

It could be...

    - unjustified and sloppy (not 'fake', but also not considered reliable evidence)
    - unjustified and malicious  (this i would consider 'fake')
    - unjustified and gamed (again, 'fake')
    - ...or just unjustified and under-specified (and would result in 'true' results if the conditions for replication were better studied and defined)

Lots of people like to think themselves smart for following 'first principles'...and then often end up falling in the same ditches. First principles + received wisdom is a bit of a contradiction... if it's 'wisdom' rather than evidence, you are skipping your principles to go with the received starting point...

The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions. Then a large number of people will points to the study, or even worse, to press characterizations of what a person thinks the study says, to support their preference on the public policy position. Immediately, anyone who has an alternate position on the public policy will be accused of being a “science denier.”

Almost no one is interested in having an honest discussion about whether or not the original paper actually says what it’s characterized to have said, and whether it was a good study in the first place.

So nowadays, when public policy is concerned, largely I disregard any scientific study that is introduced to support any position on the policy, and just do my own cost – benefit trade-off to determine my policy position.

This is a problem with the journalism and politics, it's not really about science. No scientist would trust a result that depends on a single small sample paper. Those are just stepping stones that may justify further research for more robust evidence. This fact is quite clear to scientist and it's why most would discourage the general public (including smart engineers) from reading academic articles.

But in general, I agree with you. It's ridiculous when someone pretends to shut down a complex issue by citing a random paper. However, an expert can still analyze the whole academic literature on a topic and determine what the scientific consensus is and how confident we are about it.

> No scientist would trust a result that depends on a single small sample paper.

Unfortunately, they would. There are papers with thousands of citations that don't even have data samples, just models based on assumptions.

Citations alone doesn't indicate how it is used.

Maybe a lot of scientist really liked the proposed 'model' and/or some discussion on the assumptions.

and cited it in their paper proposing some additions or follow on work.

That is also fine.

When I checked how people were citing these useless papers, almost invariably it would be in a sentence like this:

"Computational modelling is a useful technique for predicting the course of epidemics [1][2][3][4][5]"

The cited papers wouldn't actually support the statement because they'd all be unvalidated models, but citing documents that don't support the claim is super common and doesn't seem to bother anyone :( Having demonstrated a "consensus" that publishing unvalidated simulations is "useful", they would then go ahead and do another one, which would then be cited in the same way ad infinitum.

I disagree. A scientist could read a single paper and find out n is small, or identify a flaw.

But there are loads of papers like this. Then you have some literature studies which look at all these papers together and get result aggregates.

Then you get some “proper” studies which link to these aggregates, and several small studies, and you’re going to read these “proper” studies which are quoted often and deemed decent or good quality.

And at no point will you realise it’s all based on shoddy foundations.

This is for example what recently happened in social psychology

> The problem is, that as soon as a paper is published in a peer reviewed journal, it can be used to justify public policy positions.

That’s not a science problem. That’s a political problem. Making choices based on a single paper that may not be replicated is the problem.

Same here, and the more "the science" is used as a cudgel, the more people will resent it, whether or not they know a lot of it isn't even replicable.

You say you do a cost:benefit...but where do those costs come from? To me, that's just voluntarily doing your own ignorant, sloppy science I was mentioning above. If you only consider the blatantly obvious costs and benefits, you are completely ignorant of any 2nd or 3rd order...or even your own blind spots. You may radically under or overstate, or even calculate in the wrong direction.

I think a better position is that we should have a higher bar of what level of study or replication is required for a given situation...whether that be health, housing, infrastructure or whatever policy is coming in...what kind of monetary outlay and timeline of impact is expected. I don't think most people here would be happy with a 6-person study, unreplicated, deciding policy...so what IS the threshold?

I don’t analyze the papers, I analyze the policies.

Take the climate. Assuming the science is correct (storms worse, oceans rising, etc), let’s do a cost benefit analysis of the proposed policies.

The proposed policies in the US all dramatically increase the cost of energy (and therefore of everything), but only slightly slow the progression of warming, AND the bigger contributing countries in the developing world, esp China and India, will continue or increase their CO2 output.

We already know how to build dikes, and people who buy oceanfront property already know the risks, and over the last century we as a species have gotten really good as reducing deaths due to weather, so I don’t support the draconian carbon reduction proposals, and instead we can just deal with the side effects as they arise.

But how do you analyze the policies without doing science? Nothing in the above is sound to me.

"The proposed policies in the US all dramatically increase the cost of energy" - why? How do you even begin to conclude this without looking at some sort of (economic/scientific) analysis?

"only slightly slow the progression of warming" - again, how are you concluding this?

"we as a species have gotten really good as reducing deaths" - why should this trend continue? Why should it continue in the face of more extreme weather/climate change?

All I see are things you _think_ are true, and so to you your argument seems sound. But as the comment you replied to said, all I see is ignorant, sloppy science, since any meaningful analysis of these policies is by definition science. These cost/benefits you mention are not universal apparent truths.

I’m don’t need to analyze any scientific paper to form an opinion on a matter of policy; what scientific paper should I read to decide whether to support an increase on property tax in my county to support schools? The idea that I have to defer my judgment to someone in a lab coat is insulting.

As to your other questions, the answer to all (which any trivial web search would answer as I did) is “I can read”.

And I didn’t “cherry pick” some particular study or some blog that supports my preconceptions. All of the links I have provided are from sources that assume severe climate change is coming without serious interventions in greenhouse gas emissions; they are all on the “climate activist” side.

Dramatically increase the cost of energy, note that this is BY DESIGN, it’s the entire point of carbon taxes and markets: https://www.npr.org/2021/12/16/1064951646/why-the-cost-of-ca...

Only slightly slow the progression of warming: https://www.science.org/content/article/paris-climate-pact-5...

Weather related deaths down 98% in past 100 years: https://reason.org/policy-study/decline-deaths-extreme-weath...

And of course now I’m being down voted for my example.

I never challenged a single point of climate orthodoxy; I never questioned whether global warming was occurring, whether it was man-made, whether the studies and models predict warming accurately, or whether the impact will be as stated. I stipulated every single one of those points. But simply my personal conclusion that I don’t believe the policy trade-off is worth it, means that my voice is not worth hearing.

It's not the current threshold, where you just need a single paper in a semi-prestigious journal to become "the science". This was my point.

From my perspective, I don't think I ever see "the science" being decided based on a single crap paper, so it seems a bit of a skeptical straw man. Perhaps that's different in your country, state, city, county, continent, island. Perhaps that's true for a particular policy _area_ that you are invested in. In general, I find arguments on scarequotes "science" often overblown and just reinforcing existing biases.

How much 'policy' are you happy with that you've never checked to validate that it was properly replicated beforehand? How much of your skepticism is against "the science" on new policy you start with an initial dislike to...compared to policy that we consider as standard, are probably comfortable with, likely consider to be 'beneficial enough'...and yet haven't adequately scrutinized?

The COVID era has given us many rich examples of lies dressed up in a single paper.

https://www.science.org/content/article/many-scientists-citi...

And sometimes the opportunity window for studying a thing is just closed forever. Unique events are unique. That strange meteorite will not return until 300 years, or the species used in the experiment split in two, making impossible to duplicate it.

Fake is a good word. "unjustified and sloppy" is fake to me. We have different definitions of fake and shouldn't get hung up on definitions. My last word on it is I should be antagonistic against people who are deciding what goes in my son's school lunch based off of "unjustified and sloppy" studies.

First principles + received wisdom are counter balancing, not contradicting. Everything in life is a balance.

Definitions matter when it comes discussion, as what you say influences how people feel on a topic. Broad, non-specific definitions leave a lot of space for bias rather than clarity.

If you describe it as 'fake', I consider that to give the impression of 'the answer is NOT' this, and could lead to anti-policy.

If the description is 'unjustified and sloppy', that can lead to additional research to properly invalidate or potentially find something useful, so we don't throw the baby out with the bathwater.

We have different style. That's OK. Not everyone has to talk like a bureaucrat.

Be careful with this default position.

If we start from first principles: - Any one individual is not trained or experienced in the vast majority of scientific disciplines, regardless of his/her expertise in any one field - Even if half the studies published have flaws (where did you get that stat? was that study flawed?), their body of references is filtered by decades of replication and paper ranking of importance - The people doing the studies have decades of study in the particular field they're publishing about

Concluding from first principles: Its more likely that a random individual's opinion about a random niche of scientific study will be wrong than that of an individual trained on that niche

And then also starting from "received wisdom": Joe Schmuck on the street doesn't know more than scientists about their thing

The 3 things I mentioned counterbalance each other. And also notice I didn't say we should get rid of science or scientists. I said we have gone too far in the other direction of disregarding "Joe Schmuck" (or granny), which your comment demonstrates.

Also there are lots of Joe Schmucks that know better than a so called "nutritionist" that a steak is better than an Oreo. Not all Joe Schmucks, but a lot of them.

Reading many scientific studies that are done poorly, I am not convinced that you need a lot of formal training in a field to identify bad science. Many frauds and errors are blatantly visible in their data, which should have a given distribution but doesn't, or in their flawed analyses. Many others have several clearly uncontrolled variables (hint: if it's not in the paper, they didn't control it).

People of a given field will also often circle the wagons around bad actors to protect their own jobs and reputations. This introduces a very strong bias in the people who should be the most highly-informed on their own subject, and sadly makes them less trustworthy than people like Data Colada (https://datacolada.org/) who go beyond their own fields.

I did stats and applied research so I basically got to read papers as my full time job, then go home and read papers as a hobby.

Most fields are really bad at stats which is one thing that is pervasive across just about all science. More than half of science wouldn’t exist without some sort of fraud, clear selection bias, or p-hacking.

It’s to the point that one sure way to lose friends and be shunned in academia is to be good at statistics.

It’s like the plagiarism scandal, academics have been getting away with it for so long it’s become normalized. It used to be that plagiarism would be the absolute end of any academic career. Now some academics are arguing for an amnesty on the basis that everyone did it and others are ignoring it.

Edit: I should point out like the other commenter that a lot of the fraudulent data is laughably blatant. Like pulled ‘random’ numbers straight from their head, not even using any type of random number generator. Or pulling random numbers from wrong distribution. Or not using realistic noise etc.

> Its more likely that a random individual's opinion about a random niche of scientific study will be wrong than that of an individual trained on that niche

Well ... this intuition should definitely be true in theory, but in practice some fields are sufficiently distorted by biases (financial, ideological) that it's possible to be more correct as a random individual than an individual trained on a niche. The random Joe Schmuck may or may not have less knowledge, but they definitely have less career-related bias and that is often decisive.

There are plenty of examples where ordinary people manage to systematically beat the experts. A good recent example was the COVID lab leak theory. The entire field of virology and public health came down on one side of that issue, even though it was obvious that a lab accident was highly plausible, and then it turned out they all knew were all lying through their teeth and did it deliberately anyway. The rando nobodies who said "yeah, well, I don't trust the experts" came out ahead and this is now widely acknowledged. In the UK both Boris Johnson and his at-the-time health secretary have admitted that it is the most likely explanation for where COVID came from.

Ultimately, training is of no use if incentives cause people to mislead.

Wouldn't you expect that given enough experiments, you'll have some number of well done experiments that just have weird sample groups? It's well known that if you go fishing in large populations you'll find associations that disappear if you do repeated studies with different subjects. There's clearly a bias to publish interesting results, so it seems natural to expect a lot of interesting findings to not be replicable, even if the study is carefully done and there's no ill intent.

> processed foods are bad

Are they? What kind of food is not processed that we should be eating?

I think I know what people mean usually when they use this term and like that in a sentence, but this term has always confused me even far into adulthood.

Still pointing it out, since I think it's such a bad or misleading term to use.

"quite obvious but are hard to prove" ... "people love to poke holes in obvious truths, especially in well educated spaces like HN"

yes they're bad. It's not confusing, you're just overthinking it.

According to NHS:

A processed food is any food or drink that has been changed in some way when it’s made or prepared. Most foods we eat are processed in some way.

Processing can be used to:

make foods safe, for example milk is pasteurised to remove harmful bacteria make foods suitable for use, such as pressing seeds to make oil preserve foods or help them last longer, such as tinned or frozen foods change how food tastes, such as adding salt or sweeteners create ready meals and snacks

Not all processed foods are unhealthy, but many ultra-processed foods are high in calories, saturated fat, salt or sugar.

Eating too many calories, too much saturated fat, salt and sugar, and not enough fruit and vegetables and fibre is not good for you.

----

Like why should processing something make it inherently unhealthy? Do you want to eat raw meat?

> people love to poke holes in obvious truths, especially in well educated spaces like HN

Someone makes a statement that they say is obvious truth, yet it seems false or misleading. You can't get away by later saying that someone will try to counter this "obvious truth".

This is a disingenuous question because you obviously know cooked meat is not what we usually mean by processed food.

And that's why the term is confusing, because there's a magical meaning behind that, which who knows what it exactly means. And you use this term to raise children - I absolutely was confused as a child about the meaning behind it. It felt like this magical term that I was unable to comprehend. Almost like circular reasoning - if something is processed it's bad for you. What does processed mean? That it's made bad for you! It's the same thing about the word "chemical".

It's also hiding what are the exact and true health problems behind certain foods.

Then you get those terms that are used and you get the anti-terms.

"All natural", that is supposedly good, etc. Except there are quite many poisons and things in nature that can kill you.

Then people get those strong feelings associated to those labels, and they get manipulated in various directions, because something can be considered to be of that label.

It’s certainly got an element of know it when you see it. Nonetheless it’s pretty easy to figure out. One good piece of evidence to consider is whether or not process chemistry is involved in its making.

Is it - even as a child? When usually your education for the food you consume begins. I explicitly remember being confused about it.

I probably had quite a few inaccurate beliefs in general because of this type of terminology.

Another common thing that made me have inaccurate beliefs was the "E" labelling in Europe. And I remember being told that "oh look at how many 'E' (evil?)" substances are on this package.

But also Curcumin, Vitamin C, etc are considered E substances.

So just so many inaccurate ideas all around. And I don't think this inaccurate labelling has helped much if people are unhealthy.

Cooked carrot = not processed Oreo = processed Soda = processed Steak = not processed Sugar = processed

We can keep on going with all possible ingredients and all possible combinations and most sane people will agree on what is called "processed" or not.

This is disingenuous discussion dressed up in smarty pants talk.

Shall we continue or actually discuss something substantive? Like the fact that Americans are morbidly obese and eat 60 pounds of sugar a year. We look like crap.

First answer in Google says that cooked carrots and baked potatoes are processed. But I'm supposed to take your definition over what is available elsewhere? In addition frozen broccoli is processed. This is coming from Harvard Health Publishing and UCLA Health. Are they all insane?

Would you move on if I replace "not processed" with "minimally processed" and "processed" with "ultra processed"?

You're not some harbinger of truth if we do that.

In fact, it's pretty evil (idc if you're Harvard, UCLA, or a commenter on HN) to argue over whether or not you can distinguish between a cooked carrot or an Oreo.

Which brings me back to my original point. That educated spaces need a study or even just a link to an authoritative source to accept even the most basic point. Here's a Harvard link for you: https://nutritionsource.hsph.harvard.edu/processed-foods/

Hope that's satisfactory for you. LMK if you need a UCLA or Stanford link.

> Would you move on if I replace "not processed" with "minimally processed" and "processed" with "ultra processed"?

I think it would still be better to not mention "processing", because it's not about the "processing", it's about what is there in the end foods that can cause issues. So I would say that excessive consumption of foods loaded with unhealthy fats, sugars and artificial preservatives, flavor enhancers are bad for you". It's bad to eat too many fats, sugars, calories in general and not enough fruits, vegetables which contain ingredients that your body needs to properly function. If you want to use a circular word you can also just use "unhealthy foods", not "processed foods".

> In fact, it's pretty evil (idc if you're Harvard, UCLA, or a commenter on HN) to argue over whether or not you can distinguish between a cooked carrot or an Oreo.

I'm not arguing about that. I'm arguing about what is healthy and what is not. Processing food inherently doesn't make it unhealthy.

"not enough fruits, vegetables which contain ingredients that your body needs to properly function."

Is orange juice a fruit? There, take some of your own medicine lol.

In all seriousness, if you're not convinced ultra processed foods are bad for you as a rule, there's nothing I can do except throw a bunch of studies at you, which I've already argued is insane for such a basic and common sense concept

I know you probably will take great issue with "common sense", but most people I know, don't. That's why it's common lol

And presumably you've already seen lots of these studies so I don't even know why we're arguing something that makes sense and is also backed up by many studies (though a large proportion of these studies are likely to be garbage)

> Is orange juice a fruit? There, take some of your own medicine lol.

I'm not sure what you are trying to imply here? That it is? Orange juice is not a fruit, it's a juice derived from a fruit. Which is healthy in moderation, but definitely not enough alone to cover all the optimal ingredients you would get from a variance of different fruits.

> In all seriousness, if you're not convinced ultra processed foods are bad for you as a rule, there's nothing I can do except throw a bunch of studies at you, which I've already argued is insane for such a basic and common sense concept

I'm not convinced because it all depends on how it was exactly processed and the amounts.

cooked carrot = processed

oreo = ultra processed

a raw carrot and a cooked carrot have different nutritional profiles.

I literally don't know what people mean when they say processed food, and neither do they. It's a weasel word based on circular, no-true-scottsman, logic.

I claim "processed food" isn't even a cogent concept.

I claim "concept" isn't even a cogent concept.

In fact I claim "cogent" isn't even a cogent concept

How about "claim"?

See how these definitional arguments become un-useful and boring?

I know the "processed food" thing is the heap paradox because the amount of processing isn't naturally a binary, but what are your "let's just substitute a different word here" claims based on?

"When do grains of sand become a heap" is a good analogue to it.

"what are your "let's just substitute a different word here" claims based on?"

Being annoyed that even when the comment is "people love to poke holes in obvious truths, especially in well educated spaces like HN." HN commenters begin to poke holes in an obvious truth- that there even is such a thing as processed foods. It'd be like arguing that heaps of sand don't exist because it's not "cogent concept"

The space lasers guiding hurricanes along the chem-trails is totally real!

Just ask anyone in the braintrust states...

Who was second team on everest? Who was second to land on the moon?

I agree. If we relied on anecdotal evidence more often we'd have resorts on Mars by now.

"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize."

Assuming you're being sarcastic. I never said anecdotal evidence. And indeed all 3 (first principles, received wisdom, and personal experimentation) are helping get to space, arguably more than research papers

Reply on news.ycombinator.com