Want more policies based on evidence?
review.chicagobooth.edu
review.chicagobooth.edu
In terms of public policy you have to decide what you're optimizing for and that decision can't be made with data alone because it does not help resolve questions of value and fairness.
A lot of arguments about science are really arguments about confidence. E.g. most climate change scientists are fairly sure about their models, but the lack of absolutely certainty makes it possible for deniers to cherry pick a tiny collection of outlier scientists who will argue in public that it's all nonsense.
Policy makers and the media are some combination of corrupt and clueless, so they're happy to go with the false equivalence this creates.
One way to depoliticise science would be to have an international science foundation, which was funded independently of any individual government.
Of course there would be squeals of disapproval from vested interests, but that would simply highlight the problem - the vested interests don't want independent criticism or oversight. Their entire MO is based on regulatory capture which gives them the freedom (for themselves only) to operate as they want with no personal or financial consequences.
Scientific accountability would set them on the path to democratic accountability, which is the last thing they'll accept.
I think scale/proportion is also a problem. Humans seem to place a lot of value in narratives/stories but we aren't so good with quantities (e.g. https://en.wikipedia.org/wiki/Conjunction_fallacy ). Pretty much everything (economics, climate, etc.) has factors pushing it in different directions, so we can always find a counterargument to any position (e.g. we can rebuff climate change by pointing to solar cycles, CO2 causing extra plant growth, etc.); that's fine, but some factors are overwhelmingly more important than others, whilst we seem to cling on to these stories/narratives and give them more equal weighting than we should.
As a concrete example, a family member used to leave their lights on overnight, claiming that "they use more energy than normal when they're first switched on". Whilst true, the saving is cancelled out after seconds ( e.g. https://www.energy.gov/energysaver/when-turn-your-lights )
Practically no systemic analysis is done within government. At least in Chicago. Government systems are compartmentalized in ways that makes interfacing with them impossible for any worthwhile analysis. Example: the only analysis that Chicago's finance department had done on its parking tickets is a single very high level spreadsheet.
Some analysis is better than no analysis.
This is so far from being true, it undermines your point and your post.
The Federal Reserve does no systematic analysis? The U.S. Treasury does no systematic analysis? The Bureau of Labor Statistics does no systematic analysis? The Congressional Budget Office does no systematic analysis? The Centers for Disease Control do no systematic analysis?
They're actually setting other people up for disaster, hoping that they will already be gone when it hits.
I don't agree with that. Doing some analysis on a limited and possibly skewed data set can lull you into a false sense of understanding. It makes your ideas seems objective when they can in fact be completely baseless.
It's hard to believe that the percentage of successes with no analysis is GTE the percentage of successes with some analysis.
And additionally, you now have the pride/confidence thing in your even worse results because you did "analysis"...
The problem is that most in government simply are not systems thinkers. They are focused on values and fairness, and believe that once you’ve identified those values, you can directly legislate those into outcomes. This thinking leads to spectacular failures (the war on drugs, the war on poverty, tough on crime, etc).
I don't think the main problem is that "people in the government are stupid". Real life and societies are much more complex than a jet engine. There are thousands of value goals, too many to be able to put numerical targets on each one.
What is the optimal ratio of potholes to unsolved murders?
And when you forget to include any of the value goals in your model, then you get a paperclip AI scenario with terrible effects somewhere else.
Sure, but once you decide, then you should use data to optimize it. Seems pretty straightforward, and I don't think anything in this article would disagree with that.
You are also assuming we can agree on what to optimize for. In fact we do not, and will not.
You can't use data to optimize anything? Google and Facebook must be wasting their time collecting all that user data, then.
> You are also assuming we can agree on what to optimize for. In fact we do not, and will not.
We agree on lots of things to optimize for. There are cases of disagreement, but very very broad agreement as well.
No, I can use data, the problem is I don't know what all the effects of that optimization will be. So I constantly have to change what I'm optimizing for.
> We agree on lots of things to optimize for.
Broadly, but there are limited resources and each thing effects the other. So even though we agree that A and B are worth optimizing for, we will disagree on which is more important. Worse in many cases we will agree on A and B, but the data shows you cannot optimize for one without pessimism the other.
That is the broad agreement isn't really enough to do anything with, we need the details and there we disagree.
What are you trying to say here? Optimizing things with data is hard?
> Broadly, but there are limited resources and each thing effects the other. So even though we agree that A and B are worth optimizing for, we will disagree on which is more important. Worse in many cases we will agree on A and B, but the data shows you cannot optimize for one without pessimism the other.
I think we even agree broadly enough on the relative weights of many things to optimize for them. And I think we at the very least agree enough on things to partially optimize them, or pareto optimize them. In many cases there are low-hanging fruit to be picked that can optimize a metric that we all agree is good without sacrificing another.
The academy is the wrong place to direct politics, because politics already direct the academy.
From my perspective, a more likely overall improvement in the ongoing quality of policy would be a requirement that all policies sunset reasonably soon by default, even if they don't seem divisive at the time they're passed. As it regards controlled substances, this would lower the bar for repeal to "nobody particularly cares to renew it" from "nobody particularly cares to do the work to repeal it".
A lot of the most important debates of the last 100+ years has been on exactly this debate.
Rationalists like Ayn Rand (sort of), Rosa Luxemburg (red Rosa) and others on one side, and the like of Karl Popper & Hayek on the other.
When people talk about "testability" of a theory to decide if it's a scientific theory... they are borrowing from Popper's criticism of Freudianism, Marxism & the concept of metric based "government science."
It's true that for the micro decisions, like does drug X treat condition Y, medicine appears to be much more evidence-based than, say, sociology. But the overall dysfunction of medical policymaking makes me dubious of using medicine as a model for other fields of decisionmaking.
As with everything the USA sets out to do, it excels at it.
Because the harms of drinking during pregnancy are well known. Fetal Alcohol Syndrome is a real thing, as are other Fetal Alcohol Spectrum disorders.
> Evidence of harm from less than two drinks per day or 10 drinks per week is not clear
It seems there is strong evidence that "heavy" drinking is bad, the evidence that you shouldn't drink at all is much less clear.
Anecdotally, while in Europe I do see it extremely commonplace for women to have a few wines while pregnant.
[0] https://en.wikipedia.org/wiki/Fetal_alcohol_spectrum_disorde...
The number of examples of where standard of Care deviates from empirical research is staggering.
We still lose hundreds of thousands of lives a year in the US to preventable medical errors.
Many hospitals still refuse to use things like simple checklists that are proven to save lives.
I have widespread tendinitis caused by a reaction to a medication. I have seen many physicians and ALL OF THEM are just terrible with dealing with stuff they don't have expertise in.
Also, my cousin recently got diagnosed with EDS she suffered for over 20 years but they just made the diagnosis now.
I cannot comprehend why doctors aren't just open when they have not diagnosed a disease with some symptoms and ask to schedule another appointment. They go type some terms into google and at least come up with a checklist or a plan to test for diseases. The worst thing is when you have done some ACTUAL RESEARCH and can even provide certain studies but they straight up don't have the will to look at it. I think many doctors are just getting burned out by the current state of the medical system in the US, which is definitely not helping.
As a SE I often say, I have never done that before but I'm sure with a little time I can get it done. Medicine is supposed to be based on science but some doctors do not trust drug companies and studies anymore, as that is another shit show. There is pressure on researchers to obtain positive results.
One look at crime stats and this will not be a favorable idea.
Without asking "why" it may ignore That neighborhood is black, so historically racist decisions lead to extra patrolling (but no evidence of increased crime rates) which led to more arrests.
As the public doesn't understand statistics it would be easy to make that claim a truth by repeating it a few times.
Statistics are hard and the implications of some result are sometimes not intuitive. People don't care about what a standard deviation is, they just want more standard deviation than the other people over there because that makes them better.
I don't want to make a case for dismissing an empirical approach per se!
An current example from Germany right now: There's a part of the country that's in uproar because of new immigrants that came in 2015. Some people felt threatened and demanded more police in public spaces were immigrant youth gathers. So more police is present at those public spaces. Naturally this resulted in them being aware of more violent events (mostly groups of immigrant kids scuffling), which made the immigrant share of violent crimes double, and people feel "rightfully" afraid, as the stats now proof their suspicions! But the reality of course is that German kids have fights in public, too; police just didn't take note and so those fights don't end up in the data gathered.
It's not guaranteed, no. But it's the only way we have any hope of it.
> An current example from Germany right now: There's a part of the country that's in uproar because of new immigrants that came in 2015. Some people felt threatened and demanded more police in public spaces were immigrant youth gathers. So more police is present at those public spaces. Naturally this resulted in them being aware of more violent events (mostly groups of immigrant kids scuffling), which made the immigrant share of violent crimes double, and people feel "rightfully" afraid, as the stats now proof their suspicions! But the reality of course is that German kids have fights in public, too; police just didn't take note and so those fights don't end up in the data gathered.
Is that actually true, though? I'm not familiar with the figures, but it's not unreasonable to imagine there may genuinely be more violent incidents among migrants from a war torn region like that. There were certainly serious cultural assimilation problems:
https://en.wikipedia.org/wiki/New_Year%27s_Eve_sexual_assaul...
Not deadly knife fights, though. We don't want to erase the difference between brutal murder (25 stab wounds) and "a scuffle", do we?
Does that make it less of a problem?
That you brought up the murder in the first place is exactly the misuse of statistics I spoke of. About 400 people get murdered every year in Germany, and you bring up exactly the one recent case were the suspects are immigrants.
You could have pointed out that number of arrests per area is the wrong measure and why. That would have been rational. Instead, you chose to drag identity politics into it. Brilliant.
But that's what you should have said if you were making a rational argument. Instead you made it a political statement by playing the racism card.
[1] - https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-...
An elegant way to select for the most violent young men and has genuine advantages over prison.
And its really funny they use "evidence-based medicine" as a positive example. Basically this is what we should expect to happen:
>'“Evidence-based medicine has drifted in recent years from investigating and managing established disease to detecting and intervening in non-diseases,” Greenhalgh and colleagues wrote in BMJ in June.
Furthermore, the “evidence” favoring various treatments typically comes from trials in which companies decide which drugs to test, at what doses, on how many people. Often the experimental statistical “evidence” in such studies establishes a benefit for a drug that is of little practical value; the supposed benefit, while perhaps real in a mathematical sense, is so slight as to be meaningless for real patients. In other cases, perfectly sound evidence regarding a particular disease is rendered irrelevant in patients afflicted with more than one disorder (patients who are commonly seen in medical practice, but typically excluded from the trials that produced the evidence).
On top of all that, Greenhalgh and colleagues point out, the sheer volume of medical evidence makes assessing it all intelligently essentially impossible. Even the guidelines summarizing the evidence are too voluminous to be useful to doctors.
“The number of [evidence based] clinical guidelines is now both unmanageable and unfathomable,” Greenhalgh and coauthors note. In one 24-hour period in 2005, for instance, a hospital in the United Kingdom admitted 18 patients with 44 diagnoses. The relevant U.K. national guidelines for those patients totaled 3,679 pages. Estimated reading time: 122 hours.'" https://www.sciencenews.org/blog/context/evidence-based-medi...
Even if this was done correctly (it isn't, as John Ioannidis keeps pointing out), you'd get lots of false positive results (Ioannides again). The moment you look at a collection of statistics, the only available conclusion is "They all contradict each other!!1"
Until they ask physicists for help, medicine, life sciences, and many other fields will not be evidence based. We can call them "argument based", but that's not the same...
I think the problem here is that we elect governments to solve collective problems, not personal problems. Collective problems arise from the interactions of large groups of people and they are generally intractable. Even if an economist tries to accurately model human behavior, it's really hard to do as you start adding more rational agents, and then you need a theory of production and a theory of expectations formation and all of a sudden what you are doing is just a toy to be tractable.
So the zone where there is clear unambiguous science is very small, and it's generally not the zone of contention in politics. Stuff like "don't swallow arsenic" -- it doesn't really help us to govern or solve any of our political questions.
So if we can't use science to show us how to organize ourselves, the debate turns to meta-approaches which are basically philosophies of how we approach problems.
For example, a liberal might start with a Rousseau-type blank slate view of human nature and a various declaration of rights, both of which are basically statements of faith, and then try to think of some way that government can secure those rights through a period of debate, and then adopt whatever seems right to the people doing the debating, even if it means overturning long established customs.
A conservative might go -- woah, there is hidden information in customs and traditions, which we may not appreciate (it's hidden) but, as an article of faith, the fact that these customs have lasted so long means that they are probably useful for _something_, and I doubt your ability to think of every unintended consequence of your policy, so I'm going to oppose it, and instead suggest you do something much smaller, as a pilot, over here for a generation, and then see what happens.
And a libertarian might be led to say, "Your proposal infringes on my personal liberties, and thus, as an article of faith, must lead to bad outcomes."
Etc.
So the real debates we have are actually methodological debates given that we are working under imperfect information and limited knowledge about how complex interacting social systems work. Saying "we should use more science" is vacuous B.S. What we need is a better functioning public policy discourse that supports more course corrections, allows more people to air their views, and eliminates certain known distortions, like the influence of big money groups on campaigns. That's a hell of a lot more useful than pretending that scientists can be politicians or that politicians should be scientists.
A story from Chesterton's Heretics (1905):
Suppose that a great commotion arises in the street about something, let us say a lamp-post, which many influential persons desire to pull down. A grey-clad monk, who is the spirit of the Middle Ages, is approached upon the matter, and begins to say, in the arid manner of the Schoolmen, “Let us first of all consider, my brethren, the value of Light. If Light be in itself good–” At this point he is somewhat excusably knocked down. All the people make a rush for the lamp-post, the lamp-post is down in ten minutes, and they go about congratulating each other on their unmediaeval practicality. But as things go on they do not work out so easily. Some people have pulled the lamp-post down because they wanted the electric light; some because they wanted old iron; some because they wanted darkness, because their deeds were evil. Some thought it not enough of a lamp-post, some too much; some acted because they wanted to smash municipal machinery; some because they wanted to smash something. And there is war in the night, no man knowing whom he strikes. So, gradually and inevitably, to-day, to-morrow, or the next day, there comes back the conviction that the monk was right after all, and that all depends on what is the philosophy of Light. Only what we might have discussed under the gas-lamp, we now must discuss in the dark.
Similarly liberals who say things like "all men are created equal and have the same rights and can institute whatever practices seem right to them" may not know anything about Rousseau and blank slates, but they are expressing these views.
Thanks for the Chesterton quote. Chesterton is awesome. A bit of a lunatic, but also insightful and fun to read.
So, what, we should just make things up? Do you have some other decision procedure aside from the scientific method you would like to employ?
> What we need is a better functioning public policy discourse that supports more course corrections, allows more people to air their views, and eliminates certain known distortions, like the influence of big money groups on campaigns. That's a hell of a lot more useful than pretending that scientists can be politicians or that politicians should be scientists.
And what is going to inform that public policy discourse? Are we just going to talk at each other about our prior beliefs and implicit biases until one side gets tired? Or should we use the scientific method to answer empirical questions about the impacts of policy, and then debate which values we should use the knowledge gleaned therefrom to optimize? I know which one i'd prefer.
But I definitely would like to see more hard numbers in political discourse. For example, what was the effect of previous tax cuts that increased deficits? Just put out the numbers: government revenue, business investment and so on. At least people (maybe) could have a rational foundation for their opinions. But I guess I am asking too much...
But, to me, that's more of an example of money in politics and money having a big loudspeaker rather than "more science needed".
Moreover, I don't think that people who voted for Trump, voted for him because they favored corporate tax cuts, but I think it was more of a protest vote.
Of course, I could be wrong, but the problem is the outsized influence of lobbyists and the wealthy in our political system -- that was a Koch brothers job. I don't think conservatives love the Koch brothers, to be honest.
Let's remember "science" was how the economists pushed through all the banking de-regulation and liberalization of international capital flows which I am so opposed to, as well as other disastrous reforms. For example, the sadism against Greece in the EU was led by PhDs, and many PhDs pushed for more austerity.
So, for me, "more science" isn't the solution, but something like public financing of campaigns might be a better solution. I think the skepticism of technocrats is quite healthy.
I have two graduate degrees (econometrics) and while I understand and can reason with most of these models I'd argue exactly the opposite. Economics is not a "real science" or rather, it is the science of humans, and we know for sure that humans are fickle and irrational.
Economics is not a hard science yet sometimes is treated as such, and this research is then taken true as gospel (which I argue is wrong). There is a lot of things wrong with economics (I'm a fan of behavioral economics in particular) but "I want to see more data" is a non-argument. You're acting like economists are some armchair scientists waving around wet fingers in the air.
Economic decisions have to be made by politicians. That is an unavoidable fact. They can either attempt to use the best data available to optimize the values that they care about...or they can just do whatever they feel is best, informed by, what? Their gut? Their down home common sense?
Unless it's climate science, reproductive health, evolutionary biology, the existence of "electromagnetic hypersensitivity", whether fracking causes earthquakes, or a list of other topics where denialism and, yes, outright lying have arisen to challenge science that the scientific community regards as unambiguously settled.
Good read, and great take on the current stream of consciousness.