Statistics: P values are just the tip of the iceberg

It's very easy to lose focus or rush through important steps that seem insignificant. This is dangerous because bad methodology often looks a lot like good methodology but leads to very different results. When you have a precise hypothesis, it's much simpler to look at a step and say "no, that's not answering the question correctly".

It might be my own statistics-porous media filter, but it seems like this issue is finally getting the attention it deserves. Hopefully this discussion will lead to bigger and better things.

[1]http://simplystatistics.org/2015/04/29/data-analysis-subcult...

[1] http://www.amazon.co.uk/Calculus-Made-Silvanus-Phillips-Thom...

  Agreed! Much of the "problem" with P-values comes from
  them being expected to do too much; they aren't (for
  example) indicators of effect size or final proof/disproof
  of an effect's reality. And careful attention to the
  P-value can't overcome problems at the earlier steps you
  lay out here. I would differ in one spot, though: a
  P-value should not be "the last of these [inferential]
  steps" but rather closer to the beginning of critical
  thought! (More about all this here: http://wp.me/p5x2kS-Y.)

Assuming, of course, you want the truth. Lies, damn lies, and statistics as they say.

It is indisputable that a theory that is inconsistent with empirical data is a poor theory. No theory should be accepted merely because of the beauty of its logic or because it leads to conclusions that are ideologically welcome or politically convenient. Yet it is naive in the extreme to suppose that facts – especially the facts of the social sciences – speak for themselves. Not only is it true that sound analysis is unavoidably a judgment-laden mix of rigorous reasoning (“theory”) with careful observation of the facts; it is also true that the facts themselves are in large part the product of theorizing.

...

No one who reads this work – and there’s plenty of it – can possibly decide, based on the data alone, whether minimum-wage legislation does or does not reduce the employment options of low-skilled workers. It’s not just that the conclusions drawn from the empirical evidence differ wildly. It’s more that coming to a conclusion requires reasoning about - theorizing about – the data. These data (like all data) do not speak for themselves. It’s just not what data do.

http://cafehayek.com/2015/04/theorizing-about-the-facts-ther...

"Statistical research largely focuses on mathematical statistics, to the exclusion of the behaviour and processes involved in data analysis"

If there's a major problem in data collection, cleaning, or hypothesis generation, it doesn't matter whether NHST or Bayesian methods are used

"What is a p-value anyway?" -> "Doing Bayesian Data Analysis"

[I found the latter after reading one of the linked "no to p-values" articles here and I don't want to go back.]

1. Teaching people who specialise in context areas (chemists, engineers) a pragmatic way to make a decision from data in the presence of randomness.

2. Automating analyses. (remember your Benjamini-Hochberg)

I'd never actually use it myself. Jeepers, just look at the picture.

Campbell Harvey of Duke University talks with EconTalk host Russ Roberts about his research evaluating various investment and trading strategies and the challenge of measuring their effectiveness. Topics discussed include skill vs. luck, self-deception, the measures of statistical significance, skewness in investment returns, and the potential of big data.

http://www.econtalk.org/archives/2015/03/campbell_harvey.htm...