Not Even Scientists Can Easily Explain P-values | Hacker News Reader

Not Even Scientists Can Easily Explain P-values

fivethirtyeight.com

P-Values can be pretty unintuitive to people unfamiliar, but can readily be explained I think.

I do think that this book did an excellent job of explaining it:

What Is a P-Value Anyway? http://www.pearsonhighered.com/vickers/

Excerpt (starts at p.57 in the link above):

[I]f you do nothing else, please try to remember the following sentence: “the p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” Though I’d prefer that you also understood it—about which, teeth brushing.

I have three young children. In the evening, before we get to bedtime stories (bedtime stories being a nice way to end the day), we have to persuade them all to bathe, use the toilet, clean their teeth, change into pajamas, get their clothes ready for the next day and then actually get into bed (the persuading part being a nice way to go crazy). My five-year-old can often be found sitting on his bed, fully dressed, claiming to have clean teeth. The give-away is the bone dry toothbrush: he says that he has brushed his teeth, I tell him that he couldn’t have.

My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth. Or using statistician-speak: here are the data (a dry toothbrush); here is a hypothesis (my son has cleaned his teeth); the data would be unusual if the hypothesis were true, therefore we should reject the hypothesis.

[...]

So here is what to parrot when we run into each other at a bar and I still haven’t managed to work out any new party tricks: “The p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true.” When I recover from shock, you can explain it to me in terms of a toothbrush (“The probability of the toothbrush being dry if you’ve just cleaned your teeth”).

>"My reasoning here goes like this: the toothbrush is dry; it is unlikely that the toothbrush would be dry if my son had cleaned his teeth; therefore he hasn’t cleaned his teeth."

I don't think this works so well because you are jumping from a qualitative statement (dry or not dry) to a quantitative ("at least as extreme").

This is way too much for the layman and full of mathematical language. What is null hypothesis? What is "at least as extreme as"? What is probability? Nevermind that the null hypothesis forces us to think in double negatives, which isn't the most intuitive.

It also isn't clear to me how you would apply your example given how most p-values are used. I read an article that says exercising makes you richer with a 95% confidence. So, as a layman, if I attempt to apply your example, it leads me directly into a common p-value mistake: the probability of me becoming richer if I exercise. 95%, right? Wrong!

To be fair, this isn't "my" example. I selected it from the book above as a pretty good example from the book targeting p-value, the subject of this thread. It is part of a larger scheme of examples and explanations in the book. The book itself goes through and clears up things like "null hypothesis" and "at least as extreme as" using other examples.

That one quote wasn't really meant to take the layman from zero to "p-value master" in a handful of sentences.

I'm having a hard time understanding why p-values are so hard to explain without resorting to jargon... it seems very intuitive to me, and none of these examples really put it in context.

I'd explain it like this:

It's easy to lie with statistics if you take them out of context. Maybe I have an idea that drinking flouridated water causes cancer, so I take a survey and find that 40% of people who drink flouridated water develop cancer. That number is meaningless without knowing how likely it is for people who don't drink it to develop cancer, which is about 40%, so the numbers match up 1:1, completely normal. In this case, 1 is the p-value, and it gives you a starting point for developing an experiment.

>"That number is meaningless without knowing how likely it is for people who don't drink it to develop cancer, which is about 40%. That's the p-value, and it gives you a starting point for developing an experiment."

That is the frequency of cancer given not drinking fluoridated water. That is not a p-value.

Sorry, I saw how that could be misleading by saying that's so I clarified right away... not fast enough I see. I mean that comparing the observed data to a baseline gives you a p-value.

This example is still misleading because you have selected a special case where the frequencies are exactly equal. In any other case the p value will depend on the sample size.

True, but what I'm trying to convey is that if someone just wants to generally understand what a p-value is from no knowledge, understanding how to read a specific number isn't important. At first, I wasn't even going to include a specific number, but I realized not having one would be confusing in its own way.

Looking at a specific p-value and understanding its significance is only useful once they're actually looking at a study. But first they just want to know how it's derived and what it's good for.

Lest anyone skimming the headline misses it, the point here isn't that these scientists couldn't correctly explain a p-value. The scientists here all could. It's that they couldn't explain it in a way that made intuitive sense to anyone. So, rather than being a dig at scientists, I think the points here are:

1. p-values are intrinsically none-too-intuitive.

2. What p-values tell us isn't necessarily that interesting.

> What p-values tell us isn't necessarily that interesting.

The problem is that the concept sounds more important than it is if you don't understand it. This is a bad failure state.

The issue is that the probability of some phenomena happening by chance is mostly orthogonal to the probability of that phenomena being true or whatever, but this seems outside the ability of most people to grasp.

I can also say that, when I was starting stats, the term p-value seemed to hint tantalizingly at "probability." I cannot be the only person seduced into erroneously thinking that a p-value had something to do with the probability that something could be untrue.

Well a p-value is a probability, so I don't think that's exactly misleading.

> 1. p-values are intrinsically none-too-intuitive

P-values are quite intuitive. They are the ratio of observations made which agree with the null hypothesis to the total number of observations made.

Edit: If you are going to downmod, at least point out the error you perceive.

That was as intuitive as the seemless connection between the Magic Bullet Theory and Ballistics.

It comes directly from the calculation of p-values through probability theory.

I'm reminded of a joke about how Haskell empowers you to apply the infinite power of abstract mathematics using the intuitive simplicity of abstract mathematics.

I think the resistance you're encountering stems from the fact that you're appealing to formality rather than intuition.

Also from the fact that he doesn't know what he's talking about.

You talking about me?

Counting positive observations vs. total observations is a formality rather than intuition? The formality is the axioms of probability. The intuition is that you make a theory and measure how many predictions it gets right.

It's all relative, as they say.

It doesn't make sense to say in a binary sense that an observation agrees or disagrees with the null hypothesis, there are only probabilities. The p-value is the probability of seeing a set of values as extreme as your actual observations, assuming that the null hypothesis is true.

>It doesn't make sense to say...

That depends on your hypothesis.

>The p-value is the probability of seeing a set of values

That is a circular definition. What is, "probability"?

P-values are not prescriptive statements about future observations, but are descriptions of actual observations.

As TeMPOraL points out, you seem to be defining probability in general. The p-value is a very specific concept in statistics, not an abbreviation for "probability value" (https://en.wikipedia.org/wiki/P-value).

Your link defines p-value as a probability.

>More specifically, the p-value is defined as the probability of...

Edit: A good exercise would be to start with the axioms[0] of probability theory and then derive the p-value for a simple experiment, using only those axioms (and your measured values).

[0] https://en.wikipedia.org/wiki/Probability_axioms

Example: The null hypothesis is that X has a standard normal distribution, and you have one observation, with value 0.7. Since you only have one observation, your definition of p-value can only give the values 0 or 1, depending on whether you consider 0.7 to agree or disagree with the null hypothesis. The actual answer is 0.2419637 (or double that, depending on whether you do a one-tailed or two-tailed test), the probability of drawing a value >= 0.7 from a standard normal distribution (you can get this value by computing an integral on the density function of the standard normal).

You are only taking a single observation in that example, making analysis through probability theory effectively useless.

Note that, probabilities of 0 or 1 are completely allowed under the theory and the computed value in your example is still consistent with my statement and probability theory.

An issue is that probability distributions (such as standard normal) don't exist in the axioms and so must be constructed from the axioms. So, a p-value test using a standard normal distribution is taking a shortcut that hides some of the calculations, making it appear that p-value is not actually counting a ratio, however, if you look at the full calculation from the axioms, p-value represents the ratio of agreeing observations to total observations. Basically, a single observation at the level of testing a standard normal distribution may actually represent multiple observations at the level of the math required to construct the full calculation (this entirely depends on the hypothesis being tested).

Let Ω = R (real line), F = the set of Lebesque-measurable subsets of R, and P(E) = the integral over E of the density function of the standard normal. This is a probability space in the Kolmogorov sense, and my previous example carries through perfectly well in this framework. You observe x = 0.7, and compute the probability of the event X >= 0.7 as an integral. There are no ratios, and you don't need to label the observation as agreeing or disagreeing with the null hypothesis.

See my response here: https://news.ycombinator.com/item?id=10630389

You are doing it wrong. A p-value based on a single observation is meaningless. P-values are much more useful when they are continuously computed on a running experiment with sequential observations.

See the work at the LHC for an example.

FWIW, 'jsprogrammer is smuggling here the basic definition of probability - that P(event) = (number of observations in which an event happens) / (total number of observations).

That only is the case if you work on a discrete uniform distribution, which you usually don't.

The axioms are only defined on discrete uniform distributions. All other distributions must be constructed from the discrete uniform.

This is not true. If you have a discrete uniform distribution, you can only have finitely many elements in your probability space, so it wouldn't make sense to have an axiom which refers to countably-many disjoint sets. The axioms are most useful for continuous distributions, and are closely related to measure theory.

The discrete, uniform nature is obvious in the rule of computation:

Σ P(E)

Each "unit" of the computation is weighted equally (though, the value of the units may differ).

I'll give you the benefit of the doubt and assume that you aren't deliberately trolling me, but you do seem to be deeply confused. I'll just give you a link to some lecture notes that go all the way from the axioms of probability (lecture 1) to p-values (lecture 12) and leave it at that: http://www.win.tue.nl/~rmcastro/2DI90/index.php?page=lecture....

Thanks for the massive set of pages. I already went through a similar course a long time ago.

You'll note that the notes don't derive the calculation of p-value. It merely gives a trivial example (curiously, the sum of two probabilities) with a fiat interpretation.

It's curious that my last post, merely quoting the third axiom and showing its properties plain, makes you think that I am deeply confused, when you cannot even refute trivial points and must resort to arguments (poor, in that they do not address our issue here) from authority.

I assume you refer to page 28 in the first slide deck? If you would have read closely you would have noticed that it says:

> We will restrict ourselves to discrete sample spaces for now, to avoid some technical difficulties…

In the later lectures they take a look at sample spaces that are uncountably infinite. Good luck working with a sum there.

No. I am referring to lecture 12 (ie. Ch9).

What reformulation of the probability axioms are you using to eliminate the sum in the third axiom?

The axioms may be trivially extended to bounded intervals (a la calculus).

> P-values are quite intuitive.

Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1? Or do you still think that "if that were true, p-value would be entirely useless"?

In any case, you were completely wrong about p-values a few months ago; maybe they are not so intuitive after all.

https://news.ycombinator.com/item?id=10156510

Where was I wrong?

>Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1?

If the null hypothesis is true, the p-value will converge to 1. (This makes complete intuitive sense as well, if you hypothesis is true, every observation you make should/will agree with it, making the ratio of agreeable_observations:total_observations = 1.)

You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1. Perhaps for a single observation you can only get a 0 or 1 value, and, perhaps for a over/under the average test (where every experiment will give 0 or 1) your sequence of p-values for each observation would be uniformly distributed, but p-values are generally computed as a normalized sum over many observations, in which case the value should converge to 1 if the null hypothesis is true.

One unintuitive aspect is that you might expect p-value to converge to 0 if the null hypothesis is false, however p-value is undefined when the null hypothesis is false.

Ok, I see you still have your very own definition of "p-value". I just thought it was useful to make clear to the audience that your "p-values" are not the same "p-values" being discussed here.

Edit: Probably you don't care, but for the record:

"Since the value of x that defines the left tail or right tail event is a random variable, this makes the p-value a function of x and a random variable in itself defined uniformly over [0,1] interval, assuming x is continuous." [ https://en.wikipedia.org/wiki/P-value ]

"In statistics, when a p-value is used as a test statistic for a simple null hypothesis, and the distribution of the test statistic is continuous, then the p-value is uniformly distributed between 0 and 1 if the null hypothesis is true." [ https://en.wikipedia.org/wiki/Uniform_distribution_(continuo... ]

Maybe you should pull out your full derivation. Random Wikipedia text doesn't count (particularly when it is just bald assertion, as in this case).

Most null hypotheses hypothesize a normally distributed measurement error. For any given distribution (continuous uniform), of course, the p-value means something different. That is what I have been arguing this whole time.

You complained I didn't show anything, so there you have something. Do you have anything, even if it is a bald random Wikipedia assertion, supporting your position? (Rhetorical question, I couldn't care less.)

Here is a full derivation from a random blog, which I guess doesn't count either: https://shihho.wordpress.com/2012/11/27/pvalue_distribution/

Another one: https://joyeuserrance.wordpress.com/2011/04/22/proof-that-p-...

You seem awfully invested in a discussion that you don't care about.

>https://shihho.wordpress.com/2012/11/27/pvalue_distribution/

This treatment commits the same error as the other poster (evanpw; I believe nonbel commits the same). The image at the bottom is only showing p-values for single observations. That is near useless (as the chart shows). An actual experiment requires repeated observation and analysis, not one-off observations.

    npval <- pnorm( rnorm(nSim, mean, std.dev), mean, std.dev )
    hist(npval, breaks=40, xlab=xlabel, main=title, col='gray')

R is probably the worst choice of language here, but my understanding of `pnorm` is that, when passed a list of numbers as the first argument (as `rnorm` returns), it will return a list of p-values (assuming normal distribution with given parameters), where the index in the list corresponds to the p-value, taking into account only that single observation (ie. it ignores all other observations in the list [ie. not how p-value hypothesis testing works {at least, in theory; real world practices may differ}]).

Your second link appears to be the same argument (at least, they appear to use the same equations).

The charts are a nice visualization of how you can get a uniform distribution from a normal one (or vice versa) though.

You might also want to take a look at the source of the function you are relying on:

    *  DESCRIPTION
     *
     *	The main computation evaluates near-minimax approximations derived
     *	from those in "Rational Chebyshev approximations for the error
     *	function" by W. J. Cody, Math. Comp., 1969, 631-637.  This
     *	transportable program uses rational functions that theoretically
     *	approximate the normal distribution function to at least 18
     *	significant decimal digits.  The accuracy achieved depends on the
     *	arithmetic system, the compiler, the intrinsic functions, and
     *	proper selection of the machine-dependent constants.
     *

So...yeah. I hope the constants the R authors picked work for your machine!

    const static double a[5] = {
	2.2352520354606839287,
	161.02823106855587881,
	1067.6894854603709582,
	18154.981253343561249,
	0.065682337918207449113
    };
    const static double b[4] = {
	47.20258190468824187,
	976.09855173777669322,
	10260.932208618978205,
	45507.789335026729956
    };
    const static double c[9] = {
	0.39894151208813466764,
	8.8831497943883759412,
	93.506656132177855979,
	597.27027639480026226,
	2494.5375852903726711,
	6848.1904505362823326,
	11602.651437647350124,
	9842.7148383839780218,
	1.0765576773720192317e-8
    };
    const static double d[8] = {
	22.266688044328115691,
	235.38790178262499861,
	1519.377599407554805,
	6485.558298266760755,
	18615.571640885098091,
	34900.952721145977266,
	38912.003286093271411,
	19685.429676859990727
    };
    const static double p[6] = {
	0.21589853405795699,
	0.1274011611602473639,
	0.022235277870649807,
	0.001421619193227893466,
	2.9112874951168792e-5,
	0.02307344176494017303
    };
    const static double q[5] = {
	1.28426009614491121,
	0.468238212480865118,
	0.0659881378689285515,
	0.00378239633202758244,
	7.29751555083966205e-5
    };

https://svn.r-project.org/R/trunk/src/nmath/pnorm.c

Edit: Ah, a drive-by downmodder. How nice.

>"You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1."

Try this in R. It calculates p-values for testing the hypothesis that two groups (n=10) come from the same distribution (normal with mean=0, sd=1). It does 100,000 such comparisons, then makes a histogram. This results in approximately a uniform(0,1) distribution of p-values:

hist(replicate(10^5,t.test(rnorm(10),rnorm(10),var.equal=T)$p.value))

If you increase to 10^6, 10^7, etc it will converge upon the uniform distribution. That is what people mean.

Edit:

Or something like this. Keep adding to the sample, the p-value will random walk between 0 and 1. The mean will converge onto 0.5:

n=10; Nsim=10000

a=rnorm(n); b=rnorm(n)

p=matrix(nrow=Nsim,ncol=2)

for(i in 1:Nsim){

p[i,1]=length(a)

p[i,2]=t.test(a,b,var.equal=T)$p.value

a=c(a,rnorm(n)); b=c(b,rnorm(n))

plot(p[,1],p[,2],ylim=c(0,1), ylab="pvalue", xlab="sample size")

}

Especially when publication bias comes into play: https://xkcd.com/882/

Sadly true. On the one hand, the scientists ignored the multiple comparisons problem, and should have applied something like the Bonferroni or False Discovery Rate corrections. OTOH, they got publications and publicity for ignoring it.

>2. What p-values tell us isn't necessarily that interesting.

High p-value is the first thing to look at a study. If it's high, then you can simply assume the findings are unreliable. But if it's low, it doesn't yet mean anything.

The p-values allow you to discard worst mumbojumbo quickly. That might not be interesting, but it's valuable. The thing to remember is that "we don't know" =/ "untrue".

I would argue that the p-value way of reasoning (probability of something happening given a hypothesis) corresponds to an intuitive way many people reason about probability.

For example, suppose the police are investigating the robbery of a home. They find a stranger's (call him person S) fingerprints all over the house. If S claimed he was innocent, a questioner might ask: "If you're really innocent, then why were your fingerprints all over the house?" This is another way of stating that P(Data | innocent) is small (similar to the p-value concept).

Since we often don't have a strong prior reason for believing a given individual is innocent or guilty, the p-value heuristic can be reasonable.

The formal connection to P(innocent | Data) is given by Bayes' theorem

There is a third point. Which is that scientists consistently agree with clearly incorrect statements about p-values.

That means that knowing the correct definition is no guarantee that you're actually using it correctly. (Odds are you aren't.)

I've come to think that if someone can't explain something well, they probably don't understand it as well as they think. I've heard people that fancy themselves experts on this or that software concept, and when trying to explain it use all the technical jargon they possibly can. It comes off as just regurgitating things they've heard before.

In other words, I suspect the scientists that couldn't explain p-values in an intuitive way, didn't themselves fully grasp the subtleties of p-values.

My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."

I agree that makes intuitive sense when applied to explicitly constructed experiments we could attempt to exactly repeat, but the intuition begins to break down in more complicated circumstances, such as retrospective or longitutinal studies. Saying something would happen "X% of the time anyway" becomes a philosophical issue.

Yes, well, we could talk about frequentist vs bayesian statistics, but that's hardly the point of the article.

That is what a p-value means, feasibility aside. There are also more intuitive ways to think about it, "false-positive rate" being one.

Mine is remarkably similar, though less general: "if p=.05, then running a bogus experiment 20 times ensures you'll publish."

Suddenly, the abysmal state of nutritional epidemiology makes sense. How many labs do you suspect are running "red meat = death" studies?

One is reminded of an XKCD classic - https://xkcd.com/882/.

Sadly, it only goes to point out the sad state of contemporary science. Especially in terms of nutrition, the only way to stay sane is to stick to the good old street-fighting Bayes - if popularly consumed product X was leading to cancer in any way relevant to your life, you'd see people dropping like flies all over the place.

>contemporary science

I suspect you already know this, but it bears repetition, if only because it keeps us optimistic: science is a process. Even nutritional epidemiology will eventually find its way out of the abyss. :)

And yes: the ol' jellybean XKCD is quite a propos.

I agree - but sadly, the key word here is "eventually". It's little of a consolation if you'd like to use some new knowledge today.

So as it stands now, I classify every new and contemporary (younger than 10 years) study in psychology, sociology and nutrition as "not science" and "quackery" by default, until proven otherwise. I also assume that any science news reporting in any field is an outright lie with an agenda. Rarely do I see a counterexample. Seriously, science reporting does way more harm than bad research itself.

> psychology

I'd encourage you to be careful in dismissing the entirety of experimental psychology (full disclosure: I'm a cognitive neuroscientist, which is what a psychologist calls himself when he wants to distance himself from social psych).

The reproduction crisis doesn't touch all fields of psych in equivalent ways. When you reject the field as a whole, you're also rejecting stuff like this: http://cavlab.net/?lang=en

>'My go-to phrase: "Even if there were no real difference, we would see a result like this X% of the time anyway."'

Hmm, p-values are data-dependent random variables. Do this simple experiment in R where the null hypothesis of no difference is as true as can be:

replicate(2,t.test(rnorm(10),rnorm(10))$p.value)

For example, I got:

[1] 0.8446991 0.3935238

So, the p-value can vary wildly (actually will follow a uniform(0,1) distribution) even if there were no real difference.

Experimenter A would tell you that "we would see a result like this 84% of the time anyway" and experimenter B would say 39%. If enough experiments are performed, eventually someone will calculate a very small p-value. So which experimenter should we listen to? Which experimenters should bother sharing their results? All are correct.

In other words, what should be done with that information about "even if there were no difference"? How should it affect our behavior, decisions, and beliefs?

Having been both a scientist and, now, a product manager for a medical device, I think science would benefit from the work around verification vs. validation:

https://en.wikipedia.org/wiki/Verification_and_validation

A p-value is a verification tool and nothing more, and yet scientists all too often take the "hunt for the p-value" as the only goal. Rather than fish through data for a p-value that ends up defining the scientific narrative in a paper, p-values need to be integrated into a much larger validation process driven by pre-defined scientific goals. It's no wonder that so many results are found to be specious given today's scientific culture.

The point of p-values is to limit false positive rates.

If you only accept results with p-value <= 0.05, your false positive rate is at most 5%. Of course, if you had apriori knowledge that everything was a "null result" , then 100% of the results you accept would be false positives.

By Bayes' theorem, P(H|D) = P(D|H) * P(H) / P(D) . Therefore, given a fixed "prior probability" of a hypothesis P(H), a lower p-value (P(D|H)) implies a lower probability of the null being true. I think this relationship leads to a lot of confusion when thinking of p-values.

You are incorrect. False positive rates in practice appear to be about 20-50%, even with a p-value of 0.05.

I was describing the theory behind the p-value. In practice, people slice and dice the dataset in multiple ways and try several specifications which increases the false positive rate (see the multiple comparisons problem)

The best estimate right now is 20-50% replication rate for fields that rely on p less than 0.05 to judge results. So I think you meant 50-80% false positive rate, although these are not equivalent (ie is the null hypothesis of no difference ever really exactly true...):

>"Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; [...] In cell biology, two industrial laboratories reported success replicating the original results of landmark findings in only 11 and 25% of the attempted cases" http://www.ncbi.nlm.nih.gov/pubmed/26315443

Edit:

For comparison, Dr. Oz was accused of fraud and called in front of congress for only being wrong half the time: https://www.washingtonpost.com/news/morning-mix/wp/2014/12/1...

Wait, isn't P(H) probability of the tested hypothesis being true, and therefore P(D|H) probability of seeing the data given tested hypothesis is true? The p-value would be P(D|¬H), right?

Yeah I may have worded it confusingly, you can think of H as the null hypothesis in the equation (or replace it all with not H if you want)

Ok, thanks for the clarification!

Just to clarify this correct comment for others reading it: When blahblah says the false positive rate will be at most 5%, she or he means that at most, 5% of experiments will result in a false positive, not that 5% of results will be false positive results. This tripped me up when I read it the first time.

P-values are only hard to explain when you try to explain in English. Stick to the math, where it's just P(X >= x | Ho) and once you understand the math it's immediately clear in a way an English sentence (or paragraph trying to equate to the math without math) isn't. My go-to presentation on this is: http://www.biostat.jhsph.edu/~cfrangak/cominte/goodmanvalues...

In my experience, people in science who are not statisticians have not completely understood the p value. Typically people are clueless on p-Values and the statistical power and how they relate.

This is scarily true. When my lab (bioinformatics) would work/collaborate with research biologists and their labs, the research biologists lack of statistics would become very clear. Especially when we did data mining on their qpcr experiments, we constantly have to explain things like bonferroni corrections.

Edit: This is just my experience at one university with 6-7 biological research labs. I regret using the generalization, apologies.

My belief is that p-values are popular exactly BECAUSE they can be misunderstood so easily as the answer to the question people really want answered. Namely, "What are the odds that this is the right answer?"

I many years of experience in helping people understand A/B testing. And I have found that no matter how many times you give a correct explanation, people will have this exact misunderstanding. Repeatedly.

So if you're A/B testing, here is something to consider. If you successfully set up a testing culture where you decide at a 95% confidence level, and are running one A/B test per week, you will make an average of 2-3 mistakes per year. In the long run, mistakes are unavoidable. You will make mistakes. The only question is how often you make mistakes, and how bad those mistakes are.

P-values manage to bound how often you make mistakes with the trick of most of the time saying you couldn't figure out an answer. THIS IS USELESS FOR A BUSINESS. A business needs to decide what text to use, even if they're not confident.

Your A/B tests should always produce a usable business answer. The question of interest is how often you make bad mistakes and how bad they are. And p-values are useless for that.

>"I’ve come to think that the most fundamental problem with p-values is that no one can really say what they are."

Not surprising. The correct interpretation of p-values was not discovered until ~2 years ago. If you search around the internet you will find out that, according to the discoverer, he couldn't find a stats journal to publish it. The number of people who know the correct definition must be vanishingly small.

Thank god for arxiv, or I would have never understood p-values: http://arxiv.org/abs/1311.0081

tldr: Calculating the p-value + sample size is a lossless compression algorithm. The thing being compressed is a likelihood function (way of describing effect size).

I think you overestimate the importance of that paper. You can get "a" likelihood from the p-value in the same way that a sufficient statistic determines "the" likelihood function. In the best case, if the sufficient statistic is one-dimensional and you are doing a one-tailed test, both likelihood functions are the same (because the mapping from the sufficient statistic to the p-value is bijective). In general you're losing information (there will be multiple values of the sufficient statistic mapped to the same p-value).

The coin example makes perfect sense to me and I had no idea what a p-value was before I read this article.

Here's an intuitive explanation: a p-value is the probability of getting your experimental results given that your hypothesis is wrong.

I think the coin example makes sense but is not a good example in this case. You will never be able to determine the validity of a coin just by flipping it. Any sequence of flips is technically valid but possibly just very rare for a fair coin.

The p-value won't tell you if the coin is fair or not, but it can tell you the probability that the coin is fair.

If the sequence is rare enough then we do, in practice, conclude that the coin is unfair. If you can't make that conclusion then science doesn't work; no observations will convince you.

Of course. If you flip the coin 100 times and get all heads than you are safe to call the coin unfair, even though you would expect to see that result (1/2)^100 and thus would be wrong once in a while.

I think that's sort of the point that the article is making actually. That high probability does not imply truth. There are other non probabilistic ways to verify that coin is unfair, for example by looking at the density throughout the coin.

Observations do not imply truth. 0 and 1 are not probabilities, and in real life you can't prove something using observations in a logical sense. Real world runs on probabilities, not boolean logic.

Even if you look at the density of the metal throughout the coin, there's still a chance I've altered your device to report the coin is fair. Or a passing microsingularity decided to play games with the scanning beam. Or you're just imagining the whole thing.

That's not to say one should despair that the world is unknowable. One only has to get used to the fact that, in practice, "true" just means "extremely, extremely likely".

Sure, but there is a difference on the order of magnitudes between the probability that a fair coin will come up heads 100 times in a row and the probability that a microsingularity will come along and bias your results.

But yeah truth is tricky.

No no no. A p-value cannot tell you the probability a coin is fair. This is exactly the misconception that makes p-values a bad tool.

Suppose I have a coin and flip HHHHH. Can you tell me the probability the coin is fair? No, it's fundamentally unknowable. We can say that a fair coin would have a 3% chance of flipping HHHHH (the p-value), but we can't say with what probability our coin is fair.

> a p-value is the probability of getting your experimental results given that your hypothesis is wrong.

That's a common misconception. Actually it's the probability of getting your experimental results given that your null hypothesis is right.

Well normally, hypothesis is wrong <-> null hypothesis is right.

No, if the null hypothesis is right that implies that the hypothesis is wrong, but the implication relationship does not go the other direction.

Consider an experiment where your hypothesis is that cold temperatures cause the common cold. This is a good example for a thought experiment because we "know the answer" in a way (there have been a lot of experiments on this). The null hypothesis in this case is that cold temperatures are uncorrelated with incidence of the common cold.

You place people in isolation in cold areas and a control group in warm areas, and study how many get the common cold. None of the people who didn't already have colds get the common cold: because they are in isolation and the common cold is caused by rhinoviruses (which they can't get because they are in isolation), you get exactly the same results.

This disproves the hypothesis, but it does not prove the null hypothesis, that cold temperatures and the common cold are unrelated. Cold temperatures are, in fact, related to the common cold.

Try a second experiment: you place people in groups of five in cold areas and in warm areas, and discover a moderately high correlation between cold temperature and incidence of the common cold. This disproves the null hypothesis. But the simple hypothesis that cold temperature causes the common cold has also been disproven by your first experiment.

The reason for this is that the correlation between cold temperature and the common cold is a dependent correlation: given that rhinovirus is present in the system cold temperature is correlated with incidence of common cold (rhinoviruses reproduce ideally at temperatures significantly lower than human homeostatic temperature).

The null hypothesis is not just a statement that there is no independent correlation, it's a statement that there is no independent or dependent correlation. As such, the null hypothesis is an extremely broad hypothesis which is impossible to practically prove. This is why there's such a focus on finding correlations rather than finding non-correlations: you aren't going to prove the null hypothesis.

So "probability to perceive what you are perceiving just because of dumb luck"?

Then research with high p-value would just mean "we tried, and we still know practically nothing".

From my limited teaching experience, renaming "p-value" to "probability of a false positive" makes it all very simple.

This is a very common, but dangerously wrong, interpretation of p-values. Others in this thread have expressed the same misconception, see the paper linked here: https://news.ycombinator.com/item?id=10628860

Duly noted, thanks for the reference!

This is an excellent talk I attended a while ago that explains it very well from a programmer's perspective: https://www.youtube.com/watch?v=5Dnw46eC-0o&list=PL055Epbe6d...

How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true.

Or, suppose we have some data and want to use it to estimate the value of some number b. It can be that if we can assume something about b, e.g., that b = 0, then we can calculate the probability distribution of our estimate of b. This assumption about b is the null hypothesis. Intuitively it is a hypothesis that there is no effect, that what we thought might have happened didn't, was a null effect.

We can make two mistakes. We can reject the null hypothesis when it is true -- this is called Type I error. Or we can accept the null hypothesis when it is false -- this is called Type II error.

With our null hypothesis, we get and look at the distribution of our estimate of b and see where our actual estimate is in that distribution. The p-value is the probability, from the distribution of our estimate, of getting an estimate as far or farther from our null hypothesis value for b as we did. So, if the p-value is really small, say, 1%, then we can reject the null hypothesis, that is, say that it is false, and be wrong only 1% of the time.

E.g., if our null hypothesis is that b = 0 and our estimate of b is 10 and from the distribution of our estimate a value of our estimate being as far as 10 from b = 0 is 1%, then we reject that b = 0 and conclude that it b is not zero and are wrong only 1% of the time. If the probability of our estimate being greater than or equal to 10 is 1% and we reject the null hypothesis, then we conclude that b > 0.

This is all just hypothesis testing in statistics 101. Will also want to know about the power of a test, the t-test, the F ratio, the chi-squared test, and resampling and distribution-free tests.

There is more detail in

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

>"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true."

No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy."

P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf

Okay, I looked at that PDF. I read through several pages and mostly saw lots of discussion of statements the authors claimed were common and wrong. I can agree with the wrong but am less convinced about the common.

I never saw a clean definition of p-value or alpha -- probability of Type I error.

In my study of statistics, p-value and alpha are essentially the same. Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality.

The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test. Okay, but that little equation is the definition of power. So,if have some data and a p-value or alpha in mind, and have several candidate statistical hypothesis tests, say, some parametric and the others distribution-free, etc., then will want to use the test with the smallest probability of Type II error, that is, the largest beta. Okay. Now are done with beta, and no more strain or struggle needed.

A biggie point is the real role of the null hypothesis -- it lets us calculate some probabilities that, otherwise, we would not have assumptions enough to do.

If there was a fight between Fisher and Neyman-Pearson nearly 100 years ago, then I'm sorry, but now I have no sympathy for whatever the heck, if anything, they were arguing about then.

>"I never saw a clean definition of p-value or alpha -- probability of Type I error."

On re-reading this I agree. They do not make a clean comparison. They do define them though:

p-value: "the probability of the observed and more extreme data given the null"

alpha: "α is the long-run frequency of Type I errors"

To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?

Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. It is common to play fast and loose when are conditioning on something and when not.

This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha.

I strained and tried to find a difference between p value and alpha, but in current usage I see no difference. My guess is that people prefer p-value because it abbreviates probability while alpha does not.

Maybe some people want to say that

alpha = E[E[p value|given data]]

but this is not very operational. Maybe it is what people mean in which case, right, alpha is the expected value of p-value.

So, for a given experiment and hypothesis test, can draw a graph with alpha on the X-axis and beta (probability of Type II error) on the Y-axis. Then the graph shows beta as a function of alpha. So, typically the graph runs from (0,1) to (1,0) and is convex. A more powerful test is lower at each value of alpha. A perfect test is just the point at the origin. A trivial test is the one get by ignoring the real data and using a random number generator, and here the curve is just the straight line from (0,1) to (1,0) and is useful for some cases of interpolating when the available values of alpha are discrete, e.g., in resampling plans for distribution-free statistics.

Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.

>"0.001[the p value] is the conditional probability of type I error conditioned on the data"

From the paper: alpha= probability of type I error

So, there are two equations:

p value= p(alpha|data)

alpha = E[E[p value|given data]]

Substituting we get:

alpha = E[E[(alpha|data)|given data]]

What is the difference between "given data" and "data"? How do you isolate alpha?

alpha is a number. Type I error is an event.

Thinking about this more... first of all it should have been, where P(q)=="probability of q":

alpha = E[E[P(alpha|data)|given data]]

Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get:

x=f#(P(x|y)|z))

We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so

x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)}

Denote the inverse of f# as F, then:

F(x)=P(z)P(x)

Now, z was "given data" which has probability 1. So,

F(x)=P(x)

In other words, there is method to your madness.

Just routine conditional expectation. For random variables X and Y (under very meager assumptions,

E[X|Y]

is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y.

In particular

E[X] = E[E[X|Y]]

Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.

>"alpha is the expected value of p-value"

Well, you were correct about this above. You may be onto something here in terms of "actual use".

> Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it.

The probability of type I error of what exactly? Could you be more explicit?

For example: we want to perform a test at the 5% significance level. The procedure is simple: we reject the null hypothesis when p<alpha=0.05. If the null hypothesis is true, in 1000 trials on average:

- we get 0<p<0.01 10 times, we reject the null hypothesis

- we get 0.01<p<0.02 10 times, we reject the null hypothesis

- we get 0.02<p<0.03 10 times, we reject the null hypothesis

- we get 0.03<p<0.04 10 times, we reject the null hypothesis

- we get 0.04<p<0.05 10 times, we reject the null hypothesis

- we get 0.05<p<0.06 10 times, we cannot reject the null hypothesis

- we get 0.06<p<0.07 10 times, we cannot reject the null hypothesis, etc.

The type I error of the procedure is (by construction) 5%: when the null hypothesis is true, we have a rejection (false positive) on average 50 times in 1000 trials.

The "type I error rate" is a property of the test, not a property of the outcome. Using your example, we perform the test once and:

- we get p=0.001, we reject the null hypothesis. The Type I error rate of the test is 5%

- we get p=0.32, we do not reject the null hypothesis. The Type I error rate of the test is 5%

Now, could you please fill in the dotted lines?

- we get p=0.001, we [ reject the null hypothesis / do something else]. The Type I error rate of .................. is 0.1%

- we get p=0.32, we [ do not reject the null hypothesis / do something else ]. The Type I error rate of .................. is 32%

> Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality.

This is completely wrong: you're confusing the alpha and the p-value. Maybe you meant that alpha (5% in your example) is the preset type I error rate (significance level) and the p-value (here 3.14%) is calculated from the data. But conceptually the alpha and the p-value are different things and shouldn't be mixed. 3.14% is definitely not "for the data and the test, [...] the probability of Type I error".

If you're doing hypothesis testing, you calculate 3.14 which is less than 5 (alpha) and you reject the null hypothesis. If the calculation had yielded 0.69 instead of 3.14 you would also reject the null hypothesis. But you cannot consider the second result to be "stronger" than the first one. They are both below the threshold and that's all that matters.

If you're using p-values as evidential measures, the second result is indeed "stronger" because 0.69<3.14. But in this framework there are no error rates involved.

> The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test.

I've not looked at the paper carefully, but it's unlikely. Power = 1 - beta, it is not directly related to alpha. (Maybe it's just a typing error, I'm not sure what is your point. I mention it mostly for the benefit of other readers that might be confused by your definition.)

If i flipped a regular coin 1000 times, and happened to get 900 heads in a row, you may think 'hey, something is wrong with this coin' - however, could you be sure something is up? How would you go about telling whether something is really up, or if this is just a fluke chance this one time?

The motivation for the p-value is to tell you what the probability of a fluke chance this would be.

The way to do better is to run the experiment over and over, flipping this coin 1000 times again, and again. However, since science experiments are extremely expensive, instead we only run them one time, and try to 'make sure this wasnt some fluke chance'.

That it's called a "p-value" is rather indicative of it's nebulous nature. For something that is used in a rather applied context, giving it a one-letter name that has zero context is a bit of a cop-out.

I am sure it comes from probability theory, where it is common to use the notation P(x) to mean, "the probability of observing `x`".

That's how basically everything in mathematics works tho.

In order to use p-values properly, that is to make decision, you should explain what are you going to do with the information that the p-value provides. Rejecting the null hypothesis when the p-value < 0.05 is a sensible thing, but making a strong decision when the p-value is < 0.0001 doesn't make sense in many circumstances. You should have a scheme for the decision before having the p-value.

I'm surprised nobody has brought up the backlash against routine use of the p-value among scientists and journals. At least one journal has actually banned papers that report p-values:

http://www.sciencebasedmedicine.org/psychology-journal-bans-...

Leave it to psychology to ban pretty much the only scientific tool we have...

My attempt to describe it for psych research methods students - in visual novel form: http://evc-cit.info/psych018/hyptest/index.html

Could someone grade the explanation I gave my non-technical boss last week?

In the context of a regression analysis, I said, "P-values indicate the chance that the apparent effect of the variable is from random fluctuations in the data instead of the variable itself."

While we're at it, let's try this one too: The p-value, if correctly computed, is the probability that data collected under different conditions actually come from the same distribution. My definition sidesteps the word "effect."

My two cents is that once you start talking about "distributions" you add a layer of obfuscation that makes it a non-intuitive explanation.

Thanks. That's a good point. I wonder if there's a better term, or way of explaining it.

I would grade this explanation poorly, because it is wrong.

It's impossible to know the odds of an apparent effect coming from random data fluctuations or a real effect. The p-value cannot tell you these odds. The p-value only tells you one half of that---the odds that random fluctuations would cause an apparent effect.

My takeaway is to not depend on or use them wherever possible. If I do use them, I get some heuristics from a survey of a bunch of statistics experts claiming to know what it is. And I'll be sure to keep a minimal, survey size of 30. ;)

This is a surprisingly meta article... It claims there's a general lack of statistical understanding among experts, and then supports itself by making sweeping generalizations from a tiny sample size!

I'm sure that there are lots of scientists that can easily explain p-values.

"Some scientists can not easily explain p-values" should be a more accurate, and much better, title.

From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct. On the other hand, the Bayesian approach is to split the data into training and test sets, and verify how much of it holds.

Does this make sense? If the p-values are not very good at conveying "confidence in the prediction," is this another argument in favor of a more Bayesian approach to statistics? Any thoughts would be appreciated!

Well, neither your description of p-values, nor your understanding of Bayesian inference is correct, so... no?

It would be helpful to tell how the commenter is wrong, and what is the correct description.

It would also be helpful if they'd read the article.

I read the article and it's almost completely orthogonal to the questions asked by 'graffitici. There's nothing suggesting that they didn't read the article, so now I assert that you have three things to explain - where they're wrong, what's the right answer to their question and why on Earth did you think they didn't read the article?

From the article: "We want to know if results are right, but a p-value doesn’t measure that. It can’t tell you the magnitude of an effect, the strength of the evidence or the probability that the finding was the result of chance."

From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct."

If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above.

If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.

> To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above.

I disagree. There's an alternative hypothesis that could explain a comment like that - the author harbors some confused views about frequentist and Bayesian approaches to statistics. I'm inclined to believe that this hypothesis is right, because I recognize that comment as something I'd write myself back when I was more confused about this topic. Reading the article is unlikely to affect this particular issue.

> Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.

I think even one sentence explaining the gist of the author's confusion would be enough. Plenty of essays have been written on the topic, but one needs to be pointed in their general direction in order to benefit.

That's a fair point, but I'm not sure I understand the gist of the author's confusion. Any response I would craft would end up being a re-statement of the usual one sentence definition of p-values and Bayesian probability, both of which are already under discussion elsewhere in the comments.

This isn't really correct on either point. A p-value is how a frequentist establishes "statistical significance," which is itself convoluted. It has really nothing to do with validation.

On the second point, frequentists very commonly split training and testing datasets, and so this practice is pretty much orthogonal to whether you're using Bayesian or frequentist methods.

> the Bayesian approach is to split the data into training and test sets, and verify how much of it holds.

The Bayesian approach would be to say that the thing you generally want to know isn't {the probability of getting this result by chance if the hypothesis is false}, it's the probability the hypothesis is true, given everything you know up to and including the new data.

So they would calculate p(hypothesis is true given the new data) using Bayes theorem -- which requires inputting what they thought p(hypothesis is true) was at the start of the experiment, before the new data came in.

http://www.yudkowsky.net/rational/bayes is a decent explanation.

In practice, people, even scientists, treat smaller p-values as better "evidence". That's not what p-vals mean, and even if it were, there are biases in p-vals that make them problematic as the number of data points increase.

Bayesian stats aren't related to training and test sets, like in machine learning; scientific experiments rarely partition data like that. A Bayesian analysis answers the question, "Given the evidence I just collected, how should I adjust my estimate of how likely something is?" But Bayesian analyses have issues too: the biggest is not knowing what your initial, "prior" probability should be. (After all, that's usually what you're trying to find out!) With something simple like a coin flip, you have a strong prior assumption of 50% probability of heads.

And neither Bayesian nor frequentist methods address effect size. If you collect enough data, you can detect extremely small (real) effect sizes that pass statistical tests, but are still meaningless outside the context of publishing a paper. Rather than analysis type, we should incorporate more discussion of effect size, especially as the number of data points increase.

Is the video broken for anyone else?

Reply on news.ycombinator.com