Randall Munroe Responds to Gelman's Criticism of XKCD
andrewgelman.com
andrewgelman.com
Why Randall should almost apologize for a comic is a mystery to me.
I think it's a thoughtful, solidly nice and totally stand-up (and mature) thing for him to have done.
Are you sure about this? Most people don't know any statistics, and the vast majority of people who know some statistics never took it further than Stats 101. So a lot of people do blindly just apply p tests like this.
Randall's point about cancer statistics just serves to give a real world example of this, and why it's so important people better understand it.
I would be legitimately shocked if even a single doctor were willing to defend that p-test as in principle what it is for something to be statistically significant no matter what thing is at issue, rather than as a decent enough heuristic.
Nevertheless, the parenthetical you highlight is, indeed, probably literally false. I was employing a rhetorical device to convey the degree to which I found the point that, evidently, is the one Randall actually intended to be trivial.
Unless things have changed in the last twenty years, when I did a series of interviews with doctors about Bayesian probabilities in relation to an expert system project, you're going to be unpleasantly surprised at both the number who do - and the number who aren't really very sure what a p-test is at all...
I have even seen a slide presentation by consultants with a business practice of dropping all variables with p>.05, and re-estimating the model (doing this repeatedly until all variables have p>0.05).
The reliance on p-value thresholds is heartbreaking.
All in all, I think this was a wonderful exchange.
At least, 95.45% confident.
Its partly a setup for his parting joke.
Could someone point me to the "perhaps Scott Adams" comment? (is it in response to another blog entry?)
(Now I see why it's perhaps Scott Adams)
Frequentist is a special case of Bayesian. The Frequentists have much more mature tools, because they've been working on them since Gauss's day. Bayesians (especially less experienced ones) may claim that that Frequentists are old school, outdated, and don't teach undergrads the new Bayesian way of doing things.
Bayesian methods are more flexible and general, but are often slow (computationally), and can be too flexible. A Bayesian can prove anything. Frequentists have trouble eliminating some biases (because their tools aren't as flexible), but also have trouble purposely (or subconsciously) biasing their results.
I'm not going into the specifics of the methods here, just the source of their disagreements.
Frequentists are not a special case of anything. Standard frequentist arguments make absolutely no sense from a Bayesian perspective. For instance there is no prior in which such a thing as repeated significance testing errors can possibly exist. Conversely frequentists can legitimately point to a lot of things that Bayesians do which are at best highly questionable. Such as picking a default prior that gives massively high probability to clearly unlikely scenarios. (Yes the Bayesian replies, but we could pick better priors. But, the frequentist retorts, in practice you don't.)
I think that it is best to learn both, then do whatever makes most sense for your circumstances.
First of all, Bayes predates Gauss. It is inaccurate to suggest frequentist statistics predate Bayesian statistics.
Second, neither is a special case of the other.
Third, neither Bayesian methods nor frequentist methods are inherently "more flexible."
Bayesian statistics set up a model and infer model parameters by applying Bayes' rule to the data. Bayes' rule is an indisputable rule of probability.
For any model parameter b, the result of applying a bayesian model is a probability distribution for b given the available data. Criticisms of bayesian models typically center around the fact that you must use a "prior distribution" indicating the modeler's beliefs about b before seeing the data. Bayesian statisticians have a number of responses to this criticism (that some people find compelling, and others do not).
Frequentist methods build models that are justified by their properties in repeated resampling. For instance, a frequentist method is "unbiased" if, given multiple hypothetical samples, it would on average produce the correct parameter b. Frequentist hypothesis testing reports the probability of observing specified data given some assumption about b.
A standard criticism of frequentist methods is that a modeler wants a probability distribution for an unknown parameter given the known data... rather than knowing the probability of observing the realized data given some assumption about the parameter.
The truth is, I genuinely didn’t realize Frequentists and Bayesians were actual camps of people—all of whom are now emailing me.