Hacker News Reader

Comment by glial | Hacker News Reader

The article hints at this, but not publishing null results (at least in a database - somewhere!) goes hand-in-hand with the replication crisis. An experimental outcome is always a single sample from a distribution of outcomes that you would obtain if you repeated the experiment many times.

Choosing to only publish the most extreme positive values means that when the experiment is replicated, "regression to the mean" makes it very likely that the measured effect will be weaker, and possibly not statistically significant. This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science.

I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.

View on HN

There is a major issue of limited resources to replicate results, in terms of both time and funding. For example: I would assume that most important results are replicated. As a concrete example, if someone identifies a medication that (in one small trial) shows a statistically significant effect in curing some serious medical condition, then this will drive further replication attempts. On the other hand, if someone publishes a study showing that holding a pen in your mouth makes you 1% likelier to do well on the PSATs, this study will probably languish without replication for a decade — because honestly who cares? It’s basically a curiosity. I can’t help but notice that many of the headline results that characterize the “replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.

I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.

> I would assume that most important results are replicated.

GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on.

Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.

> For example: I would assume that most important results are replicated.

The example you provide is solely your assumption? Seems pretty odd to provide a baseless assumption as an "example".

>“replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.

I don't know that this is accurate. Some of these make their way to large-scale public policy, or give bona fides to people who craft far-reaching policy. This includes changes to 401k allocations to car-insurance rates and other mundane, but consequential policies.

The truth is most of science is not important outside of popular science news. So we shouldn't be surprised that the bulk of replication crises are also in the same category. Claiming this means the replication crises is not really impactful may be a case of base rate neglect.

It's also important to note that your example of medical replication is a relatively highly regulated area, where most other science is much less so.

> There is a major issue of limited resources to replicate results, in terms of both time and funding.

Undergraduate students love being involved in research. It's one of the selling points of many top universities. Grad students replicate research all the time. Maybe funding is an issue in grant fields (some research is extraordinarily expensive) but that doesn't excuse the lack of replication across the board.

I’m a researcher at an R1 and I love working with undergraduates. But it’s not true that they’re a huge source of additional resources. Quite the opposite: like any new researchers they have to be carefully supervised, which means they can actually consume a lot of time from PIs and PhD students (you can send them off on their own without supervision, but that’s how you get low quality results.) Unlike PhD students, however, undergrads tend to graduate just as they’ve finally learned the ropes. It’s still worthwhile because a few of my undergrad research interns did great work, and at least a couple of them applied to PhD programs (at least two in my lab; they’re both Assistant Professors at top CS departments.) But the answer to a resource allocation problem is almost never “here’s a simple idea that makes the resource limits go away, why aren’t people doing that.” Unfortunately.

It's kind of amazing that we discovered the scientific method, used it to invent the transistor and bring the information revolution.

Yet we still pool scientific results using only the printing press.

It's like we unlocked the tech tree but then got so caught up in chasing citations and peer review that we forgot to use the new tech we invented.

Yes, so-called "social technology" sometimes doesn't feel very advanced.

Relevant XKCD: https://xkcd.com/882/

Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.

Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.

That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon.

You need different samples. There isn't a statistical trick that gets around this.

For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1].

You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors.

You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better.

[1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.

That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.

Okay, if that's what you're saying, then I don't understand why it sounds like you're disagreeing with the poster you're responding to.

> Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive.

That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.

I really have no clue what p is, but I also really believe Randall Munroe does. Not that he's above making mistakes, but come on.

p is the chance that you would have gotten this result if the null hypothesis were true.

>For example, if 1,000,000 tests are carried out, then 5% of them (that is, 50,000 tests) are expected to lead to p < 0.05 by chance when the null hypothesis is actually true for all these tests.

https://eurradiolexp.springeropen.com/articles/10.1186/s4174....

Even if you are right, how would this response be helpful? You're not giving the right answer, you're just saying the answer given is wrong. Nobody is coming out of this interaction with corrected knowledge in their head.

This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.

> That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.

I think you're interpreting that sentence as talking about one false positive out of 20 positives? Which would be a very incorrect statement.

But it says one false positive out of 20 experiments. That's a valid statement. (Though it does depend on the proportion of negatives you're getting.)

>someone will eventually publish a false positive result without fraud.

I think most people correctly intuit that this is actually a type of very pernicious fraud.

If 20 different people all conduct the same experiment, and the 19 negative results are never published, there is no fraud involved when the 20th person publishes his positive result without realizing it is a 5% statistical anomaly. That person probably has no idea that 19 other people tried and failed at near-identical experiments.

This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?

As others already pointed out, there is no incentive to do so.

Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!"

The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.

This is completely different from one lab running the same experiment 20 times and publishing the one positive result.

It's a great xkcd but it's really wrong on one count, it's not some outside media/popular force compelling scientists to investigate something. Most of the time, researchers are looking to prove something they "know" to be true. They truly believe that jelly beans cause acne, they just need to prove it. When they get a negative result, they simply don't believe it. Something must have gone wrong, obviously, because jelly beans obviously cause acne, so maybe it's a color thing? Ah hah it's the green ones, now that we have our results we can construct other metrics to support this correct data!

Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.

Yes, I find that when reading a paper I think to myself "Do the authors really want this to be true?" and if the answer is yes as it often is, I boost my own acceptance criteria to p<.03

Particle physics still uses five sigma as the significance threshold.

> This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science.

And knowing all this, this behavy should be considered borderline scientific fraud at this point.

Richard Feynman was complaining about exactly this phenomenon fifty years ago: https://sites.cs.ucsb.edu/~ravenben/cargocult.html

>> I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.

Wait, why should PhD students do that work? That just sounds like pushing more grunt work to the lower rung of the academic hierarchy.

Nope. If you want people to do that kind of work that is important to everyone but is not directly conducive to promoting one's research career then the solution is simple: pay them.

>why should PhD students do that work?

I think there is some reasonable argument that replicating research is the first step to learning how to do good research on your own. In an ideal world, PhD students should probably be trying to replicate similar work anyway and applying existing approaches own pet problem. In practice, many gloss over this because they are narrowly focused on doing something "new" so it can get published.

PhD students are in training, and replicating a published result is a great training exercise. PhD students ARE paid. But this work won't be prioritized by their PI unless it's also a requirement of the program.

>> PhD students are in training, and replicating a published result is a great training exercise.

I just got my PhD last July and that sounds like boring drudgery rather than "good exercise". Good exercise is to have a student write their own paper, dig up their own references, formulate their own experimental hypotheses, run their own experiments, write up their own results etc. Re-doing someone else's possibly badly-done work is grunt work that should not be forced upon anyone.

>> PhD students ARE paid.

Haha. Good one XD

Not all doctoral students are paid as part of a research position.

“We thought academia was not soul crushing enough so from now on you will additionally spend 10 hours a week replicating dumb papers from 1993”

Snark aside, replication is a cornerstone of science. If someone doesn't want to be involved in science because they think it's soul-crushing, perhaps academia isn't the right place for them.

> This is not an evidence of scientific fraud...

In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. However, this distinction is not only not advertised broadly and without aversion, it is usually (in my experience) not mentioned at all, if not outright denied using persuasive rhetorical language (like, for example, when an object level instance of not achieving it is pointed to in the wild, such as in forum conversations). This may not be fraud (that requires intent I think?), but it achieves the same end: misinforming people.

I absolutely agree with your database idea, and if science would like me to take them seriously (something near how seriously they take themselves) they'd also have to go much further.

[1] Not unlike in religion, a competing metaphysical framework (model of reality) to science.

> Not unlike in religion, a competing metaphysical framework (model of reality) to science.

No. Correlation fallacy.

Fallacy fallacy.

Naive Realism fallacy.

This sort of comment is why I think a lot of philosophy is just communicating poorly to make yourself sound smart.

In your footnote, for example, you translated your philosophy-speak into English (metaphysical framework -> model of reality). Why not just say that? Your entire comment goes into "philosophy mode" and communicates a few very simple ideas in overcomplicated language.

Science and religion are pretty poorly understood as competing models of reality. Religion originates when people make up answers to other people's questions to gain social standing, and religion continues due to (among other things) anchoring bias--the bias people have toward continuing to believe what they already believe. While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief.

You said:

> In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1].

This is presented as some sort of gotcha, but it's not: few scientists will claim that science is being practiced perfectly or even well. Outside of a few areas such as particle physics, we're quite aware that our ability to practice scientific ideals is hampered by funding, publication incentives, availability of test subjects in human studies, data privacy, etc. And we're aware that this means that our conclusions need to be understood as probabilities rather than 100%-confidence facts.

There are certainly some people who treat scientific conclusions with religious absolute confidence, but doing that is fundamentally against scientific principles. The accusation you are leveling against science would be better targeted toward people: generally science journalists and the science-illiterate public rather than scientists themselves. The entire reproducibility crisis is scientists using science to show that our practice of science is too imperfect to result in high-confidence conclusions.

Religious people jumping on the replication crisis because they think it disproves science is rich. The replication crisis isn't a disproof of science, it's an application of science. The reason we know that there's a replication crisis is because scientists asked "How confident can we be in the conclusions of existing studies?" and applied science to answer that question. If you really think science is invalid, then you can't use science to prove that.

And the fact remains that any confidence in conclusions at all is more than religion has to offer, because again, religion isn't trying to model reality--the fact that religion produces a model of reality is merely an unfortunate side-effect.

> While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief.

A problem: you're talking to one right now, and you (your mind's model of reality, technically - you do not have access to the state of the things you claim to) could hardly be more wrong.

From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?

https://en.wikipedia.org/wiki/Theory_of_mind

https://en.wikipedia.org/wiki/Direct_and_indirect_realism

I can appreciate that this approach may seem unworthy of anything more than a rhetorical response that dodges the question (and the importance of the phenomena these links discuss), so hopefully you can take the challenge seriously. I am more than happy to offer a more substantive reply later, but if you declare victory by fiat[1] it's a bit tough to have a serious conversation.

[1] Roughly: declaring that one's opinion of the unknowable is necessarily correct, and that it is(!) contrary to my (actual) stance.

> A problem: you're talking to one right now, and you (your mind's model of reality, technically - you do not have access to the state of the things you claim to) could hardly be more wrong.

A problem: you're talking to someone who used to be religious, so I have as much access to the internal thinking of a religious person as you do.

A second problem: people's self-perceptions of their own internal processes are quite often measurably wrong.

> From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?

> https://en.wikipedia.org/wiki/Theory_of_mind

> https://en.wikipedia.org/wiki/Direct_and_indirect_realism

Short answer: not in any interesting way.

Long answer:

From large quantities of experience, I would guess that you're about to make a special pleading argument that based on convenience beliefs that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.

Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.

Let's cut to the chase with a relevant parable:

The Buddha and his disciple were walking down the road. Suddenly, the disciple drew his sword and cut the Buddha in half at the waist. The Buddha turned to his disciple and said, "Now you're beginning to understand!"

Would you be willing to reproduce this parable experimentally with you as the Buddha? After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal, that will be just fine, right?

The thing is, science is perfectly capable of answering this question--it's not unknowable. The experiment of cutting someone in half with a sword has sadly already been performed too many times in history: we don't need to perform it again. The scientific answer, which we already have, is that no amount of changing our perception prevents the person cut in half with a sword from dying in horrible agony. And when you're not speaking philosophese, you already believe the scientific answer just like every philosopher who believes perception is reality until faced with the prospect of being cut in half with a sword. So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?

My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.

More parts of philosophy I think we can discard without losing anything of worth:

1. Arguing that perception=reality when it's convenient and refusing to practice it in any other context.

2. Talking about the unknowable as if we know it.

> A problem: you're talking to someone who used to be religious, so I have as much access to the internal thinking of a religious person as you do.

Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.

> Short answer: not in any interesting way.

To you, or to all people?

> From large quantities of experience, I would guess

At least here you realize you're guessing.

> that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.

How could you possibly know this?

> Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.

Here you are speaking as if you perceive reality as it is directly, no interpretation, no bias.

> Let's cut to the chase with a relevant parable:

> After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal...

It may be relevant, but it certainly isn't representative.

> ...that will be just fine, right?

Incorrect. Might you be thinking in binary?

> The thing is, science is perfectly capable of answering this question--it's not unknowable.

"Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.

> The experiment of cutting someone in half with a sword...

Look at you milking this incorrect strawman!

> you already believe

You can not read my mind. You are literally hallucinating.

> So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?

Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.

> My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.

Note that this itself is an opinion (an opinion of an opinion).

It is also a tautology, which in this case I would classify as cheating (you are making claims to know the unknown, and backing it up with a tautology).

> More parts of philosophy I think we can discard without losing anything of worth

When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.

> More parts of philosophy I think we can discard without losing anything of worth:

> 2. Talking about the unknowable as if we know it.

Ok, this is too much. Have you been putting me on this whole time?

I'll shoot the fish in a barrel in your followup later on (I am "posting too fast"...what on this site isn't at best a half truth?).

1. If you don't want me to speculate about your point, make your point. I'm not going to participate in you trying to "teach" with the Socratic method of a bunch of leading questions that try to make answers you don't like impossible.

2. I said I was guessing, and it's possible my guess war wrong. But frankly, nothing you've said actually indicates that my guess of your point was wrong--it simply indicates that you didn't understand my post. So I'll ask you point blank: do you believe perception is reality? A simple, "yes" or a "no" will suffice, no need to flower it up with philosophese.

3. You say you're a religious person. Which religion? I doubt you believe the vague, slippery, undefined religion that exists only to allow religious people to debate atheists without having to defend the absurd beliefs of their actual religion.

> Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.

No, I'm saying my access to the mind of a religious person is as good as yours.

> > The thing is, science is perfectly capable of answering this question--it's not unknowable.

> "Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.

Yep. Which is why I didn't say "science is perfectly capable of answering ALL QUESTIONS". What I said was "Science is perfectly capable of answering THIS QUESTION." Since you're intent on taking everything out of context, I will insist on clarifying that what I mean is, "Science is perfectly capable of answering the question of whether perception is reality or not--whether perception is reality or not is not unknowable."

I will go on to say, religion isn't capable of answering any questions. No criticism of science actually gives any validity to religion.

> Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.

Your invisible friend is whatever deity your religion believes in. You said you were religious.

> When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.

Bro, what are you even asking? If you want to know why I think philosophy is often poor communication presented as intelligence, it's crap like the above quoted paragraph.

No, I'm not going to pseudo-code out the definition of the word "think", if that's what you're asking me to do.

I think I've demonstrated my point adequately.

What point? You haven't even made a point, let alone demonstrated it.

You say you "think" you've made your point. Remember when you said:

> When you say "think", I'm curious what you mean.

Strange that you suddenly know what that word means.

At least you are consistent!

One thing I'll add to my other post is:

Anything you're going to say to try to show that religion is trying to model the world is going to be an attempt to use logic and observation, because those are the tools that work, and you know those are the tools that work. You're just using them with an end goal in mind due to aforementioned anchoring bias, which isn't really the way to use them.

What you probably aren't going to do is tell me to pray and God will tell give me the answers, or tell me to read a book written by ancient people who believed stuff like "the sky is a dome", because you're well aware nobody takes that seriously as a way of modeling reality. I mean, maybe you are going to tell me to do those things: if so, that's embarrassing.

And the latter is what religion has to offer for modeling reality that science doesn't have. Sure, religion uses logic and observation because they so obviously work, but science has logic and observation too, and frankly, is better at it--we don't need religion (or philosophy) to tell us how to logic or observe.

So if you really want to present religion as a way to model reality, that's what you've got to prove has value for modeling reality: talking to invisible friends and consulting outdated writings. We're already convinced that logic and observation work: how do your weird additions compete with that?

Reply on news.ycombinator.com