Hacker News Reader

Comment by peatmoss | Hacker News Reader

Lest anyone skimming the headline misses it, the point here isn't that these scientists couldn't correctly explain a p-value. The scientists here all could. It's that they couldn't explain it in a way that made intuitive sense to anyone. So, rather than being a dig at scientists, I think the points here are:

1. p-values are intrinsically none-too-intuitive.

2. What p-values tell us isn't necessarily that interesting.

View on HN

> What p-values tell us isn't necessarily that interesting.

The problem is that the concept sounds more important than it is if you don't understand it. This is a bad failure state.

The issue is that the probability of some phenomena happening by chance is mostly orthogonal to the probability of that phenomena being true or whatever, but this seems outside the ability of most people to grasp.

I can also say that, when I was starting stats, the term p-value seemed to hint tantalizingly at "probability." I cannot be the only person seduced into erroneously thinking that a p-value had something to do with the probability that something could be untrue.

Well a p-value is a probability, so I don't think that's exactly misleading.

Especially when publication bias comes into play: https://xkcd.com/882/

Sadly true. On the one hand, the scientists ignored the multiple comparisons problem, and should have applied something like the Bonferroni or False Discovery Rate corrections. OTOH, they got publications and publicity for ignoring it.

>2. What p-values tell us isn't necessarily that interesting.

High p-value is the first thing to look at a study. If it's high, then you can simply assume the findings are unreliable. But if it's low, it doesn't yet mean anything.

The p-values allow you to discard worst mumbojumbo quickly. That might not be interesting, but it's valuable. The thing to remember is that "we don't know" =/ "untrue".

I would argue that the p-value way of reasoning (probability of something happening given a hypothesis) corresponds to an intuitive way many people reason about probability.

For example, suppose the police are investigating the robbery of a home. They find a stranger's (call him person S) fingerprints all over the house. If S claimed he was innocent, a questioner might ask: "If you're really innocent, then why were your fingerprints all over the house?" This is another way of stating that P(Data | innocent) is small (similar to the p-value concept).

Since we often don't have a strong prior reason for believing a given individual is innocent or guilty, the p-value heuristic can be reasonable.

The formal connection to P(innocent | Data) is given by Bayes' theorem

There is a third point. Which is that scientists consistently agree with clearly incorrect statements about p-values.

That means that knowing the correct definition is no guarantee that you're actually using it correctly. (Odds are you aren't.)

I've come to think that if someone can't explain something well, they probably don't understand it as well as they think. I've heard people that fancy themselves experts on this or that software concept, and when trying to explain it use all the technical jargon they possibly can. It comes off as just regurgitating things they've heard before.

In other words, I suspect the scientists that couldn't explain p-values in an intuitive way, didn't themselves fully grasp the subtleties of p-values.

> 1. p-values are intrinsically none-too-intuitive

P-values are quite intuitive. They are the ratio of observations made which agree with the null hypothesis to the total number of observations made.

Edit: If you are going to downmod, at least point out the error you perceive.

> P-values are quite intuitive.

Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1? Or do you still think that "if that were true, p-value would be entirely useless"?

In any case, you were completely wrong about p-values a few months ago; maybe they are not so intuitive after all.

https://news.ycombinator.com/item?id=10156510

Where was I wrong?

>Do you agree now that if the null hypothesis is true the p-value is uniformly distributed between 0 and 1?

If the null hypothesis is true, the p-value will converge to 1. (This makes complete intuitive sense as well, if you hypothesis is true, every observation you make should/will agree with it, making the ratio of agreeable_observations:total_observations = 1.)

You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1. Perhaps for a single observation you can only get a 0 or 1 value, and, perhaps for a over/under the average test (where every experiment will give 0 or 1) your sequence of p-values for each observation would be uniformly distributed, but p-values are generally computed as a normalized sum over many observations, in which case the value should converge to 1 if the null hypothesis is true.

One unintuitive aspect is that you might expect p-value to converge to 0 if the null hypothesis is false, however p-value is undefined when the null hypothesis is false.

Ok, I see you still have your very own definition of "p-value". I just thought it was useful to make clear to the audience that your "p-values" are not the same "p-values" being discussed here.

Edit: Probably you don't care, but for the record:

"Since the value of x that defines the left tail or right tail event is a random variable, this makes the p-value a function of x and a random variable in itself defined uniformly over [0,1] interval, assuming x is continuous." [ https://en.wikipedia.org/wiki/P-value ]

"In statistics, when a p-value is used as a test statistic for a simple null hypothesis, and the distribution of the test statistic is continuous, then the p-value is uniformly distributed between 0 and 1 if the null hypothesis is true." [ https://en.wikipedia.org/wiki/Uniform_distribution_(continuo... ]

Maybe you should pull out your full derivation. Random Wikipedia text doesn't count (particularly when it is just bald assertion, as in this case).

Most null hypotheses hypothesize a normally distributed measurement error. For any given distribution (continuous uniform), of course, the p-value means something different. That is what I have been arguing this whole time.

You complained I didn't show anything, so there you have something. Do you have anything, even if it is a bald random Wikipedia assertion, supporting your position? (Rhetorical question, I couldn't care less.)

Here is a full derivation from a random blog, which I guess doesn't count either: https://shihho.wordpress.com/2012/11/27/pvalue_distribution/

Another one: https://joyeuserrance.wordpress.com/2011/04/22/proof-that-p-...

You seem awfully invested in a discussion that you don't care about.

>https://shihho.wordpress.com/2012/11/27/pvalue_distribution/

This treatment commits the same error as the other poster (evanpw; I believe nonbel commits the same). The image at the bottom is only showing p-values for single observations. That is near useless (as the chart shows). An actual experiment requires repeated observation and analysis, not one-off observations.

    npval <- pnorm( rnorm(nSim, mean, std.dev), mean, std.dev )
    hist(npval, breaks=40, xlab=xlabel, main=title, col='gray')

R is probably the worst choice of language here, but my understanding of `pnorm` is that, when passed a list of numbers as the first argument (as `rnorm` returns), it will return a list of p-values (assuming normal distribution with given parameters), where the index in the list corresponds to the p-value, taking into account only that single observation (ie. it ignores all other observations in the list [ie. not how p-value hypothesis testing works {at least, in theory; real world practices may differ}]).

Your second link appears to be the same argument (at least, they appear to use the same equations).

The charts are a nice visualization of how you can get a uniform distribution from a normal one (or vice versa) though.

You might also want to take a look at the source of the function you are relying on:

    *  DESCRIPTION
     *
     *	The main computation evaluates near-minimax approximations derived
     *	from those in "Rational Chebyshev approximations for the error
     *	function" by W. J. Cody, Math. Comp., 1969, 631-637.  This
     *	transportable program uses rational functions that theoretically
     *	approximate the normal distribution function to at least 18
     *	significant decimal digits.  The accuracy achieved depends on the
     *	arithmetic system, the compiler, the intrinsic functions, and
     *	proper selection of the machine-dependent constants.
     *

So...yeah. I hope the constants the R authors picked work for your machine!

    const static double a[5] = {
	2.2352520354606839287,
	161.02823106855587881,
	1067.6894854603709582,
	18154.981253343561249,
	0.065682337918207449113
    };
    const static double b[4] = {
	47.20258190468824187,
	976.09855173777669322,
	10260.932208618978205,
	45507.789335026729956
    };
    const static double c[9] = {
	0.39894151208813466764,
	8.8831497943883759412,
	93.506656132177855979,
	597.27027639480026226,
	2494.5375852903726711,
	6848.1904505362823326,
	11602.651437647350124,
	9842.7148383839780218,
	1.0765576773720192317e-8
    };
    const static double d[8] = {
	22.266688044328115691,
	235.38790178262499861,
	1519.377599407554805,
	6485.558298266760755,
	18615.571640885098091,
	34900.952721145977266,
	38912.003286093271411,
	19685.429676859990727
    };
    const static double p[6] = {
	0.21589853405795699,
	0.1274011611602473639,
	0.022235277870649807,
	0.001421619193227893466,
	2.9112874951168792e-5,
	0.02307344176494017303
    };
    const static double q[5] = {
	1.28426009614491121,
	0.468238212480865118,
	0.0659881378689285515,
	0.00378239633202758244,
	7.29751555083966205e-5
    };

https://svn.r-project.org/R/trunk/src/nmath/pnorm.c

Edit: Ah, a drive-by downmodder. How nice.

>"You haven't shown anything where a true null hypothesis uniformly generates p-values between 0 and 1."

Try this in R. It calculates p-values for testing the hypothesis that two groups (n=10) come from the same distribution (normal with mean=0, sd=1). It does 100,000 such comparisons, then makes a histogram. This results in approximately a uniform(0,1) distribution of p-values:

hist(replicate(10^5,t.test(rnorm(10),rnorm(10),var.equal=T)$p.value))

If you increase to 10^6, 10^7, etc it will converge upon the uniform distribution. That is what people mean.

Edit:

Or something like this. Keep adding to the sample, the p-value will random walk between 0 and 1. The mean will converge onto 0.5:

n=10; Nsim=10000

a=rnorm(n); b=rnorm(n)

p=matrix(nrow=Nsim,ncol=2)

for(i in 1:Nsim){

p[i,1]=length(a)

p[i,2]=t.test(a,b,var.equal=T)$p.value

a=c(a,rnorm(n)); b=c(b,rnorm(n))

plot(p[,1],p[,2],ylim=c(0,1), ylab="pvalue", xlab="sample size")

}

It doesn't make sense to say in a binary sense that an observation agrees or disagrees with the null hypothesis, there are only probabilities. The p-value is the probability of seeing a set of values as extreme as your actual observations, assuming that the null hypothesis is true.

FWIW, 'jsprogrammer is smuggling here the basic definition of probability - that P(event) = (number of observations in which an event happens) / (total number of observations).

That only is the case if you work on a discrete uniform distribution, which you usually don't.

The axioms are only defined on discrete uniform distributions. All other distributions must be constructed from the discrete uniform.

This is not true. If you have a discrete uniform distribution, you can only have finitely many elements in your probability space, so it wouldn't make sense to have an axiom which refers to countably-many disjoint sets. The axioms are most useful for continuous distributions, and are closely related to measure theory.

The discrete, uniform nature is obvious in the rule of computation:

Σ P(E)

Each "unit" of the computation is weighted equally (though, the value of the units may differ).

I'll give you the benefit of the doubt and assume that you aren't deliberately trolling me, but you do seem to be deeply confused. I'll just give you a link to some lecture notes that go all the way from the axioms of probability (lecture 1) to p-values (lecture 12) and leave it at that: http://www.win.tue.nl/~rmcastro/2DI90/index.php?page=lecture....

Thanks for the massive set of pages. I already went through a similar course a long time ago.

You'll note that the notes don't derive the calculation of p-value. It merely gives a trivial example (curiously, the sum of two probabilities) with a fiat interpretation.

It's curious that my last post, merely quoting the third axiom and showing its properties plain, makes you think that I am deeply confused, when you cannot even refute trivial points and must resort to arguments (poor, in that they do not address our issue here) from authority.

I assume you refer to page 28 in the first slide deck? If you would have read closely you would have noticed that it says:

> We will restrict ourselves to discrete sample spaces for now, to avoid some technical difficulties…

In the later lectures they take a look at sample spaces that are uncountably infinite. Good luck working with a sum there.

No. I am referring to lecture 12 (ie. Ch9).

What reformulation of the probability axioms are you using to eliminate the sum in the third axiom?

The axioms may be trivially extended to bounded intervals (a la calculus).

>It doesn't make sense to say...

That depends on your hypothesis.

>The p-value is the probability of seeing a set of values

That is a circular definition. What is, "probability"?

P-values are not prescriptive statements about future observations, but are descriptions of actual observations.

As TeMPOraL points out, you seem to be defining probability in general. The p-value is a very specific concept in statistics, not an abbreviation for "probability value" (https://en.wikipedia.org/wiki/P-value).

Your link defines p-value as a probability.

>More specifically, the p-value is defined as the probability of...

Edit: A good exercise would be to start with the axioms[0] of probability theory and then derive the p-value for a simple experiment, using only those axioms (and your measured values).

[0] https://en.wikipedia.org/wiki/Probability_axioms

Example: The null hypothesis is that X has a standard normal distribution, and you have one observation, with value 0.7. Since you only have one observation, your definition of p-value can only give the values 0 or 1, depending on whether you consider 0.7 to agree or disagree with the null hypothesis. The actual answer is 0.2419637 (or double that, depending on whether you do a one-tailed or two-tailed test), the probability of drawing a value >= 0.7 from a standard normal distribution (you can get this value by computing an integral on the density function of the standard normal).

You are only taking a single observation in that example, making analysis through probability theory effectively useless.

Note that, probabilities of 0 or 1 are completely allowed under the theory and the computed value in your example is still consistent with my statement and probability theory.

An issue is that probability distributions (such as standard normal) don't exist in the axioms and so must be constructed from the axioms. So, a p-value test using a standard normal distribution is taking a shortcut that hides some of the calculations, making it appear that p-value is not actually counting a ratio, however, if you look at the full calculation from the axioms, p-value represents the ratio of agreeing observations to total observations. Basically, a single observation at the level of testing a standard normal distribution may actually represent multiple observations at the level of the math required to construct the full calculation (this entirely depends on the hypothesis being tested).

Let Ω = R (real line), F = the set of Lebesque-measurable subsets of R, and P(E) = the integral over E of the density function of the standard normal. This is a probability space in the Kolmogorov sense, and my previous example carries through perfectly well in this framework. You observe x = 0.7, and compute the probability of the event X >= 0.7 as an integral. There are no ratios, and you don't need to label the observation as agreeing or disagreeing with the null hypothesis.

See my response here: https://news.ycombinator.com/item?id=10630389

You are doing it wrong. A p-value based on a single observation is meaningless. P-values are much more useful when they are continuously computed on a running experiment with sequential observations.

See the work at the LHC for an example.

That was as intuitive as the seemless connection between the Magic Bullet Theory and Ballistics.

It comes directly from the calculation of p-values through probability theory.

I'm reminded of a joke about how Haskell empowers you to apply the infinite power of abstract mathematics using the intuitive simplicity of abstract mathematics.

I think the resistance you're encountering stems from the fact that you're appealing to formality rather than intuition.

Counting positive observations vs. total observations is a formality rather than intuition? The formality is the axioms of probability. The intuition is that you make a theory and measure how many predictions it gets right.

It's all relative, as they say.

Also from the fact that he doesn't know what he's talking about.

You talking about me?

Reply on news.ycombinator.com