Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
I mostly agree with this, but you never have that in a difficult decision.
> Another issue is that science is expensive and we need to make decisions all the time whether there is any science backing them or not. So what do you do if there's no science that meets the minimum standards?
Right! My entire point is that maybe data driven decision making isn't as useful as we think because it mostly doesn't hit scientific levels of accuracy. Maybe there are other, more intuitive systems, that make as much if not more sense, in absence of good data. How would we even know either way? We can't!
Regardless I'd love for the discussion to be "this is the best we can do" rather than "yes we did A/B testing so we know it's true!!!"
> I mostly agree with this, but you never have that in a difficult decision.
Consider the problem "should we cure congenital deafness in infants?"
To Deaf Community Leaders, this isn't a difficult decision at all; they are strongly against curing deaf children because it makes their power base smaller.
To deaf parents of deaf children, this is a tricky choice. Curing the child's deafness dramatically improves its prospects in life, but it also inevitably cuts the child off from the parents' community. You have to choose between how good you want your child's future to be, and how close you'd like your relationship with them to be.
To hearing parents of deaf children, this is again a trivial choice; obviously you'd cure the child.
BUT, curing deafness is definitely (1) a large effect that is (2) easily reproduced.
A couple of things might make a decision difficult:
- Some course of action will produce a big effect. Would that be good or bad?
-- Ok, assume there's a good effect out there to be achieved. Is it worth the cost of obtaining it?
Those questions, variously applied, occupy a lot of people's time and brainpower. In the curing-a-deaf-child example, the parents are making a tradeoff between quality of their child's life (which is good), and closeness to their child (also good). Paying one for the other means giving up something good, which makes the decision nontrivial. But this is a difficult decision because the effects are large, not because they're small.
In contrast, if you're dealing with an effect of very small size, or one that can't be reproduced ("lose weight by following our new diet!")... this might feel like a difficult decision, but it shouldn't. It does not matter what you decide, because (by hypothesis!) your decision will have no effect on the outcome! (Or, for the "very small effect size" case, at most a very small effect.)