The High Cost of Not Doing Experiments (2015) | Hacker News Reader

The High Cost of Not Doing Experiments (2015)

behavioralscientist.org

94 pointsgwern31 comments

I'm not sure if this is really plausible, but I would love to see a legislative rule instituted which stated that programs, regulations, and spending measures cannot be passed without doing the following:

* Clearly laying out the goals that bill sponsors are trying to achieve with the proposed change as well as the reasoning driving the change

* Clearly articulating a set of metrics by which we can judge the efficacy of the program or change

* Defining a period in which studies or review will be conducted to evaluate efficacy and who will be funded to do them

If the change is later reviewed and it is found to be ineffective, it should either be forced to be modified or immediately rolled back. This would protect against a whole host of legislative silliness such as pork belly / vote buying projects, bill scope creep, the immortality of improperly validated programs as mentioned in the article, and knee jerk reaction regulations after tragedies or in the face of anecdata and moralizing.

There might be an interesring underlying problem here which you are trying to solve, but not addressing explicitly. Perhaps humanity is, for the first time, entering a period where it is possible for corruption to go un-checked. If you look at history, go back a bit and it could be described as just groups of people teaming up to kill each other. Over history, the trend was generally towards larger groups. Now the group is nearly as large as possible (global), and there's no threat to the group. So the interesting problem is that those with power over the group now have no incentive to keep the group's evolutionary fitness level high--maybe their incentive is even to decrease the group's fitness? Looking at examples in the article and in the comments, it's hard to believe this is never the case. We know the War on Poverty, Drug War, War on Terror, and other efforts have intentionally been designed to not solve the problems they were tasked with solving (poverty, drug use, and terror have not measurably dropped due to those efforts, despite trillions spent).

So when you say it "should be" you are advocating perhaps a difficult path for humanity. I personally would like to see humanity try to go down that path of improving our evolutionary fitness over our environment, but maybe to do that we need to recognize this inconvenient truth. I don't see how these things you mention change, otherwise.

Youtuber CGP Grey has a related video about this reality of leadership: https://youtu.be/rStL7niR7gs

I’ve been saying exactly this for years. Want to start a party that only advocates for candidates and laws that follow these rules? We could call the platform/mechanism “test-first” legislation.

While this sounds like a good idea, and has me nodding in agreement, it has a potential downside.

A somewhat analogous example is during the Reagan administration, they announced a rule that no new environmental regulations could be passed unless they performed an exhaustive cost-benefit analysis. It turns out that such an analysis is impossible. It was rather transparently recognized as a way to deliberately cripple the EPA.

A less analogous yet instructive example of government trying to regulate itself: Most states have "balanced budget" laws at the state level, yet most states run deficit budgets.

For those curious, in practice, one of the problems we find with this kind of thing in practice is the phenomenon of policy-based evidence:

https://en.wikipedia.org/wiki/Policy-based_evidence_making

That is to say, governments in charge of deciding goals, metrics and studies/reviews will inevitably cherry-pick, pay consultants, and fund those things which selectively support their desired policy-implementations and outcomes.

You may say: "well, make the thing independent", but that's easier said than done.

You might also say: "well, it will be immediately obvious if anyone is doing this", to which i'd respond that it really isn't immediately obvious to most people, and if the efficacy of policies/metrics/evaluations were immediately obvious, we would often negate the need to be attempting/implementing the additional overhead of formally introducing evidence-based policy (since we'd commonly agree on policy and whether it worked).

Which is of course not to say that evidence/experiments is not noble or the right thing to do, but more so to point out that governance is a far tricker situation than just saying "well, we'll fix it by legislating that governments have to do the right thing..."

Like kweinber, I've also advocated this for years. I'll accept that it's hard, but I won't accept that it's impossible. Think of it as "scientific government." We institute falsifiable hypotheses and test them. This can only be done by folks who want the right answer however, and not too many people in government are like that. They may want their answer to be right, but of course that is not the same thing.

At the very least, set up a criterion (or criteria) at which we will agree that a policy has failed.

Betsy DeVos is being disruptive in U.S. education. Will it work? Take the US rankings now. If in four years our ranking has fallen, we try something else.

Try a tax cut. If the economy is x% worse 18 months after it going into effect, it failed.

I'm sure when we first start this we will struggle to get the metrics right and to ask the right questions. But we should aim in that direction.

I suspect that the reason we don't has to do with money, and in a bad way.

I call it "evidence based government" to match evidence-based medicine. But it is one of the things I'd most like to see happen, because the real world seems like far too small an influence on politics nowadays.

Evidence-based works too, but I chose scientific because of the falsifiable hypothesis approach. We set it up so that we can eliminate things that don't work from being retried under similar conditions just because a different party takes over.

> But in fact D.A.R.E., as it has been conducted for the past thirty years at least, doesn’t decrease children’s use of drugs.

That's an understatement. Numerous studies have found that D.A.R.E. increases drug use:

https://en.wikipedia.org/wiki/Drug_Abuse_Resistance_Educatio...

Another problem I have with DARE is that it conflates some important messages (like don't give in to peer pressure) with its message of all drugs will ruin your life. When people learn that certain drugs they mention are just not that bad, other important messages could be dismissed as well.

Note: I don't have any study to support this but it's my hypothesis.

Exactly this. The program is infantilizing and not based in fact. There are great reasons why you may not want to do a number of drugs: opiates (high addiction potential, incapacitating); meth (wrecks you); cocaine (less problematic but you can't trust the cutting agents, it can exacerbate cardiac conditions, it can give you transient ED if you're a guy); molly (one poor formulation can kill you or leave you mentally incapacitated for life). I don't recall this ever being reviewed at a D.A.R.E event. All I remember hearing is "drugs are categorically bad and dangerous" which as you say falls down under scrutiny when you think about alcohol, weed, or even cocaine to some degree.

Alcohol, weed and cocaine are pretty bad and dangerous, though. It just happens to be that those are more or less socially acceptable ways to become temporarily stupid at the expense of long-term health, which I agree makes "abstinence only" education efforts seem hypocritical. Since people are going to self-medicate one way or another, teaching responsible use seems like the better choice.

Weed and certain cocaine uses aren't bad.

I partake but there are absolutely negative side effects to both. Cocaine is pretty damn addictive and weed is hardly consequence free either (although it probably has the best accessibility:side effect ratio out of any drug that i'm aware of)

If by "not bad" you mean "not as bad as other drugs" or "I can live with the consequences", then sure. Otherwise, messing with your body's chemical pathways tends to have negative side-effects, especially when you overdo it.

https://en.wikipedia.org/wiki/Cannabis_(drug)#Adverse_effect...

https://en.wikipedia.org/wiki/Cocaine#Adverse_effects

In my experience, weed has tended to make me spacey and forgetful. Unless it's headbanger skunk, in which case there's more agitation. And after 50+ years, perhaps some of that has become permanent. Or it could have been all the LSD, or just garden-variety dementia :)

But cocaine, that's some iffy stuff. It's possible to get addicted, even snorting hydrochloride salt. And once you get into freebasing aka crack, you're also looking at ~permanent downregulation of dopamine receptors. So basically, there's no more feeling happy :( Without the cocaine, anyway.

Most of us who went through the program regard it defiantly. I don't think I've seen anyone wear a DARE shirt in a way that wasn't ironic

> That's an understatement. Numerous studies have found that D.A.R.E. increases drug use

...which helps the prison-industrial complex (in California at least) and the DEA. This makes me wonder...

From the linked page: "Students who enter the program [...] are informed [...] about the government's beliefs about the dangers of recreational drug use [...]"

This wording is very suggestive, and I think that part of the problem might be 'the government's beliefs'.

When I was studying abroad in the US I was surprised by the lack of nuance in the information provided by the university. It stated that 1. Drunk sex equals rape (but only if you are male) 2. Drinking alcohol can lead to pregnancy (without any elaboration).

I found it very hard to take any drug or alcohol-related information from the university serious after reading that. You feel like a little child that's being lied to, and I can see how this increases drug abuse. The zero-tolerance policy for drugs and alcohol under 21 also feels very distopian, and is enforced in a very dogmatic way, without much common sense.

The problem we have with the social sciences is that we don't have good general theories to make sense of the experiments. Think of Newton's laws of motion. We could do a few experiments and calculate g and frictional coefficients and then we could know the answers to a whole class of problems. We don't have to do experiments on how fast a x kg block will slide down an inclined plane for every value of x.

Having a theory also enables us to get more out of a single experiment. Think about how the Michelson–Morley experiment led to Einstein's theory of relativity and all it's implications.

We don't have any such framework for the social sciences. We have trouble generalizing from the results of one experiment to another experiment with the situations slightly altered.

To some extent I agree with you that behavioral science/psychology needs a unifying theory. Attempts at this have been made and are still alive and well. For example, take a look at Radical Behaviorism [1]--the philosophy of science behind (applied) behavior analysis.

The overarching goal of behavior analysis is to treat the study of behavior as a natural science. That is, to identify orderly relations (i.e., laws) between behavior and the environment.

[1]: https://en.wikipedia.org/wiki/Radical_behaviorism

I accept your argument from the perspective that it would allow social scientists/other interested individuals to assess new data in an easy framework. But, I don't think it would change the political problem that the author is describing.

Some (admittedly weak) examples illustrate this point: despite our knowledge of physics and our exploitation of this knowledge in the form of aircraft and rockets, flat-earth believers and those who don't believe/understand acceleration abound.

We know the mechanisms by which vaccines work - and could go observe the diseases they prevent by simply taking a trip, or reviewing public health data from different countries. But, anti-vaccine belief is also increasing.

There are many more of these stories, where the 'easy' idea beats out the well-established scientific consensus. Thus, I'm not sure that a fundamental understanding of how influence/addiction/personality work would change political support for these ineffective programs.

Flat earthers have zero influence on the development of physics, and their ideas don't matter to anyone except themselves and a few opportunistic journalists.

This is not true in the social sciences where it's very hard to do physics-grade research, especially given that the social sciences are very politicised, and research is often done to prove a political or moral point.

So in practice our understanding of personal and social/political psychology is pre-Copernican. There's a lot of moralising - which you will agree or disagree with, depending on your predisposition - but very little high quality research into the way that moral and political decisions are made individually and collectively.

We're left with a mythology of free objective democratic choice in politics which is clearly naive and reliably breeds monsters, but as yet there isn't a better model of choice to replace it.

I agree with your point - as mentioned in my first line, I state that such a comprehensive model would be very helpful for academic purposes.

I don't believe that it would change popular opinion - indeed, flat-earthers and other 'anti-intellectual' fads continue despite not being accepted by serious researchers.

I had an analogical vision while reading your comment. It looks like this: 1. Hard Science -- poking a rock with a stick. 2. Soft Science -- poking a random stranger on the bus with a stick. HTH

Well, that's not actually true at all. There are very simple hypotheses motivating things like DARE or Scared Straight, for example. What happens is, you do an experiment, and those simple experiments don't work out the way you thought. And so you have to explain it, which requires a bit of work.

This isn't any different from other sciences, where you have one hypothesis, test it, and it doesn't go the way you thought, and then people are left scratching their heads for a bit, until someone guesses right and demonstrates it experimentally.

Those mathematical models in physics you mention we accept because of debates over which was correct, for example, followed by experiments (and observations!). You see gambles on explanations of unexplained results in molecular biology all the time that don't really involve the same level of mathematical precision.

I think you're overstating the extent to which scientific process occurs in all disciplines with very precise mathematical models with low stochastic error. As the complexity of your system increases, the residual noise in any given model, especially models with little established information, will be high.

What works clearing house https://ies.ed.gov/ncee/wwc/ seems like a step in the right direction. If there are more such efforts to summarize research(especially to address the problem of reproducibility in Social Sciences and Micro Economics) that would be awesome.

I think at the root of the problem is statistical ignorance, combined with confused morale judgment, along the lines of "but if we think X is effective, it would be wrong to withhold providing X for everyone equally!"

Finland now has a government program for doing such studies, called Kokeilun Paikka ("Place to Experiment").

They are using it to do a random trial of basic income. The results to date are ambiguous, showing both that it's hard to get a large n and hard to pick the measurement criteria.

Also: although I agree with the thrust of the book, there are additional issues for programs like Head Start: 1> it's hard sometimes to find or identify the kids who need it most, so by making it broader you improve your chances of reaching those most at need, at the cost of "helping" others who may not really need it 2> when you make programs universal, you get broader support than when you only extend it to those most in need. Look at the additional restrictions just placed last month on those in the US getting public assistance -- restrictions which make it hard for those most in need to actually get the benefit! This is why the New Deal extended social security to everyone and dressed it up it so that it appeared to be an insurance scheme you paid into: everyone would feel they "earned it" so nobody would want to cut it.

Thus in the US school meals are not universal or free, which stigmatizes the kids who get them. In most developed countries the kids all get lunch in school, which helps the nutrition of the poorer kids. Likewise, making public transport free would pull cars off the road, making drivers happier...but when they feel they are paying for "those people", voters typically hate using road taxes to pay for streetcars and subways. Perhaps the ability to tax self-driving vehicle fleets will change that.

This sort of goes back to the Finnish observation of the difficulty of making measurements.

Needs a [2015] tag.

Updated. Thanks!

Reply on news.ycombinator.com