TLDR: People tend to make policy decisions based on data even when the data is basically useless, I'm more worried about that then blindly following good data.
Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
I mostly agree with this, but you never have that in a difficult decision.
> Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
Right! My entire point is that maybe data driven decision making isn't as useful as we think because it mostly doesn't hit scientific levels of accuracy. Maybe there are other, more intuitive systems, that make as much if not more sense, in absence of good data. How would we even know either way? We can't!
Regardless I'd love for the discussion to be "this is the best we can do" rather than "yes we did A/B testing so we know it's true!!!"
> I mostly agree with this, but you never have that in a difficult decision.
Consider the problem "should we cure congenital deafness in infants?"
To Deaf Community Leaders, this isn't a difficult decision at all; they are strongly against curing deaf children because it makes their power base smaller.
To deaf parents of deaf children, this is a tricky choice. Curing the child's deafness dramatically improves its prospects in life, but it also inevitably cuts the child off from the parents' community. You have to choose between how good you want your child's future to be, and how close you'd like your relationship with them to be.
To hearing parents of deaf children, this is again a trivial choice; obviously you'd cure the child.
BUT, curing deafness is definitely (1) a large effect that is (2) easily reproduced.
A couple of things might make a decision difficult:
- Some course of action will produce a big effect. Would that be good or bad?
-- Ok, assume there's a good effect out there to be achieved. Is it worth the cost of obtaining it?
Those questions, variously applied, occupy a lot of people's time and brainpower. In the curing-a-deaf-child example, the parents are making a tradeoff between quality of their child's life (which is good), and closeness to their child (also good). Paying one for the other means giving up something good, which makes the decision nontrivial. But this is a difficult decision because the effects are large, not because they're small.
In contrast, if you're dealing with an effect of very small size, or one that can't be reproduced ("lose weight by following our new diet!")... this might feel like a difficult decision, but it shouldn't. It does not matter what you decide, because (by hypothesis!) your decision will have no effect on the outcome! (Or, for the "very small effect size" case, at most a very small effect.)
It seems as though you'd missed the "only" in the clause. Further, they say nothing w.r.t. "does not pass" (aka: not hitting minimum threshold). Both your complaint and their point are orthogonal.
P value alone does not tell you whether you have good data. It just tells you how well a particular model fits the data that you have. It won't (and can't) tell you if your data set is missing data that would alter the P value were it to be included.
P value alone is not enough to say "good" or "bad." That's why this statement matters.
Put another way: tons of stuff in prod dev gets done without p-values entirely because it's the first time you have any data at all about something. I'm questioning whether this is hugely valuable and obviously better than using my intuition or not, since we really don't have a good understanding of the baseline.