Hacker News Reader

>Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic

Eh, not really much prestige, but perhaps lots of press.

The big thing is to not take any single paper as "scientific truth" like what you read in a textbook. The scientific literature is not a textbook, it's the place where science's self-correction happens. Things in text books have been battle-tested, and widely accepted usually due to replication. And even then, textbooks have errors and perpetuate some falsehoods! Any individual scientific paper will not even likely be replicated, because it's working on something that did not turn out to be fundamental to the path of science.

Scientific papers in the literature are how scientists share their small steps while exploring the n-dimensional world. That n is very high, so there are lots of directions to explore, and most of them will not be that interesting. If it is, somebody else may decide to take a look too. If it's a hot-topic, then many many people will. Scientists have to operate in a world where they are exploring what is and is not true, and their relationship with uncertainty is strong; it's absolutely essential to embrace uncertainty.

The biggest thing is to distrust news reporting about science. It's correct about 1% of the time. The motivations of a reporter trying to somehow make an interesting article by deadline on a topic they don't understand are far more distorted than the scientists publishing a paper they expect 30-50 people to ever read. And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse.

> And that's if you actually get a real reporter to look at these things; most articles are just press releases that are cut and pasted from a university press lackey into the newspaper's publishing system. Those are usually even worse.

Actually if the researchers expect the paper to be high impact they will often provide the press release to the University's press department themselves - these types of press releases are merely a layman directed abstract and tend to be reasonably OK.

This raises two interesting points.

1. Sometimes the lowest quality newspapers have the best quality article on a given publication.

2. Why aren't all research publications required to be accompanied by a second abstract written in plain low jargon language that can be understood at some level by someone with high school science?

I can't quite agree, from my (albeit limited) observation, the copy-paste hype from press offices is almost always the source of ridiculous overstatements that make people distrust science. The press office's job is to make the university look as good as possible, and when they play the game of telephone with the researchers, they tend to overhype far more than a disinterested reporter would. MIT press office in particular is terrible at this, as are many lesser-known but aspiring schools.

I think maybe it depends a little bit on which university it is: perhaps a small university will not have enough money to hire a big dedicated PR office, so they will rely more on materials submitted directly by the scientist and stick closer to reality. But the PR offices of big universities (in particular MIT) put out pure hype which is very precariously connected to reality. A lot of work goes into these press releases (they will send someone from the PR office to interview the researcher and so on), but the work is not really aimed towards providing an accurate summary of the research...

As an MIT graduate I sadly concur with this statement.

To #2, because even putting aside how many scientists would have no interest in doing that, they would also be generally unqualified. Being a good researcher doesn’t make you a good teacher, and you need to be a very good teacher to turn your results into something a high school student could understand, and yet which still communicates your results accurately.

You're absolutely right here. Many journals do #2, and I generally find these to be worthless. For example, the Author Summaries on PLoS articles in a field that I'm not familiar with almost never help me. Often times when I read a paper in detail and journal club it with others, the conclusions that we derive from the data presented in figures has quite different emphasis than what is emphasized in the paper itself.

Sometimes with hindsight one can go back to a paper and say "this is important because of X" but it can be really hard to do that at the beginning, and often times the true value of the data may be different than the hypothesis under which the data was originally generated.

> 2. Why aren't all research publications required to be accompanied by a second abstract written in plain low jargon language that can be understood at some level by someone with high school science?

Because the target audience of research publications is other researchers in the same field; while outreach to nonscientists is fine, it's simply not the goal of a research paper.

Besides, for many papers this is impossible and/or pointless. My research results are incremental results in a specialized subfield of mathematics. I don't know how to explain them in anything but the vaguest terms to anyone without a reasonable background in that subfield, and I also don't know why such a person would care about them.

> I find it extremely difficult to trust new research. Any tips for reversing this mindset? I know it's not too great to be feeling this way.

Why is it not great to feel this way? You give a bunch of good reasons why it is okay to feel as you do.

In general you should ignore any new research until it's had a large well run meta-analysis of it.

Until then, it's all "we think" and "maybe". And that's ok, unless you're forcing people to take a treatment or depriving them of their liberty based on the research.

I share your concerns, yet can't think of a solution. I think the only thing that can be done is to factor in a large amount of doubt, and amortize that doubt as more definitive studies come in over time.

>"After learning about the perverse incentives for publishing scientific papers ..."

Can you be more specific, what was the source or your learning? Was there a particular piece or pieces you read? If so would you be able to share them?

I'm a statistician and I read medical studies and experimental studies pretty much daily. My advice on a simple rule of thumb that is likely to separate out quality research: are the data and statistical code made available? Another, are if the study hypotheses and design are pre-registered. The Cox model is VERY popular, but you have to ask two things: is the hazard reasonably proportional? If you thought learning to code meant you had to be careful about mistakes and typos, statistics is another layer of a similar size.

> Toxoplasmosis in particular seems intuitively like the perfect combination of new research plus massive prestige for whoever publishes papers on the topic

While I absolutely agree with the general sentiment, some of the findings related to toxoplasmosis are well-replicated and supported by meta analysis, especially the link to schizophrenia and reaction time.

Flegr was mostly ignored for a decade and his findings are only now reaching mainstream recognition, after replication by independent teams.

Some of the recent "toxoplasmosis linked to X" papers are indeed of low-quality (low sample size and worse).

the biggest problem is the reliance on linear models and their interpretations.

its easy to have something statistically significant but yet produce a shitty prediction. they then take this horrible model and use its coefficients to make ridiculous claims, like doing X makes you Y times more likely to Z.

thats great for a business wanting to find ways to make money, but when this crap gets in the government level these crappy models are used to legislate us.

What would it mean to trust or believe in a theory? What finishing line in our minds does it cross beyond which we accept it as proven or regard it as confirmed?

I don't think there is one. We are fallible and even our best theories are tentative and provisional. When we do rely on ideas it only because we know no better alternatives.

Listen, the science presented is better than no science. I think you are conflating published peer reviewed articles with the incentives of Academic Institutions, Journals and publishing houses. In my view it's unfair to let the scientists be the antagonist in this debate.