learnfromerror

Why I attended a conference on "Fixing Science"sponsored by the National Association of Scholars.

Karen Kafadar, the ASA president, would like the ASA to clearly distinguish itself from the ruling recommended by its Executive Director. She says he was only wearing his "author" hat when he declared he was adding a rule to the ASA 2016 guide. That rule is: never use the word "significance", and never use any P-value thresholds (e.g., .05, .005) in interpreting data. The ASA has so far not make this qualification and the lawyers are having a heyday.Is she right?

The ASA's P-value project: why it's doing more harm than good https://errorstatistics.com/2019/11/14/the-asas-p-value-proj...

P-value thresholds: forfeit at your peril. https://errorstatistics.files.wordpress.com/2019/11/mayo-201...

The ASA's latest recommendations to journals is to ban using "significance" & "significant", & never use any pre-designated p-value in interpreting results. The result is that there can be no tests & no falsifications even of the statistical kind.

If you were on a committee to draw up guidelines on P-values and replication, what's the first definition you'd check? Exactly. They didn't. It's wrong throughout the National Academies of Science recently released book.

Might I mention my book (Statistical Inference as Severe Testing: How to Get Beyond the Science Wars (Mayo CUP)). https://twitter.com/learnfromerror/status/107197020637171302... I did not know of Hacker news but was trying to trace why my blog errorstatstics.com got its maximal # of hits in over 7 years as a result of my posting their letter on the ASA Statement Now I'll disappear.

I think it is the way they blithely mention some statisticians prefer to use other methods, with a list of examples, that suggests they are blessing them. Surely these other methods ought to be scrutinized, we don't know they wold detect irreplication as significance tests do. The big issue is really all about using frequentist methods with biasing selection effects, multiple testing, cherry-picking, data-dredging, post hoc subgroups, etc. Only problem is that many who espouse the "other methods" declare that these data-dependent moves do not alter their inferences. Some are vs adjustments for multiplicity, & even deny error control matters (This stems from the Likelihood Principle.) If you consider the ASA guide as allowing that (in tension with principle 4 vs data dredging when it comes to frequentist methods) then the danger the authors mention is real. What was, and is, really needed is a discussion about whether error control matters to inference.

The authors of the letter use “induction” to mean what I call a probabilism. Here probability is used to quantify degrees of belief, confirmation, plausibility support–whether absolute or comparative. This is rooted in the old idea of confirmation by enumerative induction. Conclusions of statistical tests go strictly beyond their premises, and it does not follow that we cannot assess the warrant of the inference without using a probabilism. They are qualified by the error probing capacities of the tools. A claim is severely tested when it is subjected to and passes a test that probably would have found it flawed if it is. The notion isn’t limited to formal testing, but holds as well for estimation, prediction, exploration and problem solving. You don’t have evidence for a claim if little if anything has been done to probe and rule out how it may be flawed. That is the minimal principle for evidence.

Popper spoke similarly of corroboration only he was unable to cash it out. He wasn’t sufficiently well versed in statistics, and anyway, he wanted to distinguish corroboration from induction, as the latter was being used at the time. The same impetus led Neyman to speak of inference as an act (of inferring). I explain all this in my recent book, Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars (Mayo 2018,CUP. As I say there,

“In my view an account of what is warranted and unwarranted to infer – a normative epistemology – is not a matter of using probability to assign rational beliefs, but to control and assess how well probed claims are.” (p. 54)