This year’s Nobel prizes prompt soul-searching among economists
economist.com
economist.com
The article seems to motivate criticism of RCTs in only the barest manner: "what of external validity?" No shit, Sherlock, which is why the credibility revolution also argues for frequent and cross-contextual replications. "RCTs present ethical quandaries in re denying the control group the benefits of a presumed good intervention". Again, duh. The problem is these presumed good interventions often aren't which is why the most resounding impact of RCTs is not proving treatments successful beyond our wildest dreams, but rather finding a series of disappointing and in some cases devastating null effects. Bringing up deworming is especially rich in light of the serious concerns raised by deworming RCTs about the value and effect.
It is also weird to talk about the credibility term solely or entirely in terms of RCTs. As economists often say RCTs are the gold standard, but there exist many silver standards and causal inference writ large and a broader concern with well identified pseudo-experimental or observational work has also developed thanks to the same forces and same people credited here.
Thank god for Esther Duflo. Oh, and her husband is a half decent economist too, I guess.
https://www.edx.org/bio/esther-duflo
Amazing I can type that. The net ain't all bad.
Do astrophysicists use controlled trials to study the universe?
It doesn't have much to do with theory building at all, as most theory involving QM doesn't need to concern itself with exactly why QM works despite its strange philosophical meaning.
As a generous guess at most 40 of these models are wrong, since they all describe basically the same few events with vastly different mechanisms. I guess that majority of interesting but wrong models is what you are after?
As for trials, the universe itself has provided billions upon billions of different objects to perform measurements on.
But that aside, what I find disturbing is the economic couching of basic human questions: do you need economic long-term effects that can be proven to provide deworming to poor children? Is that what we have arrived at - wellbeing is not even a concern. A fictional economic greater good is the measure of success, the health and comfort of poor people is not?
I'd say there is really a need for soul-searching among economists - to rethink why capitalism has brought so much property but also leaves so many people in poverty, and why it has taken such radical forms that in the richest country on earth thousands starve and tens of thousands die of treatable medical issues - and millions more in a global view.
Those who want to make practical improvements quickly with those limited funds available (from developing country governments, foreign aid, charity, etc.) need to make effective use of those funds. I don't think it is a stretch to say it is a moral obligation to make sure those funds are put to good use.
Doing RCTs is quite key to identifying the most effective approaches whether the goal is lives saved/$, incremental years of education/$, malaria cases reduced/$, etc. With that data, you can direct funds to the most efficient cause in any given area.
Duflo & Banerjee are quite modest about any judgement on whether the focus should be on improving quantity or quality of life and how one should measure quality of life. Their books are quite clear that one's one view of preferences doesn't necessarily line up with those who you are trying to help.
And one last point, yes, of course, their work would go through ethics review panels. Basically anything with human subjects does and their work would obviously qualify.
Totally agree that Esther Duflo is fantastic and that the contributions of these people go beyond just RCTs, e.g. Esther Duflo's work on the returns of education in Indonesia.
It kinda reveals the deep underlying assumptions; "of course we can know the results of these interventions in advance, so we must base our ethics on that"... ok, but if you already know the effects, why are you testing them? You're testing them because you don't know. Unfortunately, there's no royal road to science knowledge, and, yeah, that means that some people are going to fail to get positive interventions, and some people are going to get negative interventions. Either that, or nobody gets anything and we just keep blundering on in ignorance. There is no answer where nobody takes any risks and everybody gets the good stuff guaranteed.
At the end of the lecture, a young student at the back of the room timidly asked, “Do you have any controls?” Well, the great surgeon drew himself up to his full height, hit the desk, and said, “Do you mean did I not operate on half the patients?” The hall grew very quiet then. The voice at the back of the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the visitor’s fist really came down as he thundered, “Of course not. That would have doomed half of them to their death.”
God, it was quiet then, and one could scarcely hear the small voice ask, “Which half?”
https://www.lesswrong.com/posts/Jvi9LLcvZxm529496/rationalit...
There is a really brutal dynamic involved. It is hard to be uncertain and make a decision. However, when dealing with problems more complicated than assembling a sandwich, it is nigh impossible to be certain about anything.
We all know software so here is a software example - if I walk up to a new computer, fire up a web browser and go to news.ycombinator.com, will it work? Probably, but there are many things that could go wrong - configuration, hardware, new bugs, old bugs, etc. But I can't afford to worry about any of that because I only have 8 hours in a day and I have to just assume it will work out. Usually it does. If it doesn't, then I start exploring what the system is doing and why.
But this happens with everything and quietly trains people to approach complicated situations with great confidence and pick up the pieces if (and only if) someone or something flags that there is a problem. This approach doesn't work for policy but there is a constant influx of people who think this way getting into positions of power and making policy decisions.
It is a fact that these questions of evidence are well known and indeed ancient, but there is constant and heavy pressure on culture to backslide and stop accepting that there is uncertainty that needs to be controlled. Making decisions while being uncertain is hard and by-and-large has to be learned. Even learning it as a skill is hard because it is about how we behave more than about what we know. Figuring out how to embed those behaviours at culturally at scale is a component why the Enlightenment was a big deal.
Which is to say this seems neither analogous to medicine or ethical in the same fashion to controls for medicines of unknown efficacy. Maybe, if somehow they had no choice but their "phased roll-out", they have a justification but otherwise, what were they even considering? Not treating people?
See: https://www.betterevaluation.org/resources/example/Primary_s...
It seems intuitively obvious, not "maybe, somehow", that bringing medical treatment to thousands of rural communities doesn't happen instantly.
We didn't send the man to the Moon on the basis of statistical experiments, we knew it was possible even before the first screw was put in on the rocket.
I am fearful that depending on experiments would lead to the same dire credibility issues that plague experimental psychology.
There are all sort of models still widely believed today. Some are unassailable (evolution, germ theory), some seem right more often than not ("don't eat bacon"), and some seem so right but nothing based on them works ("cholesterol is bad").
So this sort of naive belief in our ability to model the processes in a body or a cell has out of fashion at least since WW2. It's still used to come up with ideas to try. But even there, purely random exploration of the search space is not consistently worse.
Now when you get into whole body or things with mutations such as cancer I agree with you while heartedly. While we can understand in a petri dish how to kill something in a human you've got to worry about things like delivery, hidden single cells, toxicity, and drug proliferation all of which can be different in every single person with a cancer. In biotech we're still just barely out of the dark ages.
To bring this to the original topic, we're still in the dark ages with fields like economics. While we understand many of levers that exist we have little idea of how and when to effectively pull them.
But even if you are sure you got the model right, you still need a long time study to prove it. The current discussion on e-cigs are mostly about informing people that they are taking a risk because there are unanswered questions.
Putting a man on the moon was a lot of risk and there were many unknowns and I am sure the pilots knew that very well. Gladly, they still did it.
> experimental psychology
had precisely the problem that their patients often didn't really have a choice in that matter. So I think the field deserves its place. They might need to shed a lot of hubris before they can be taken seriously again. I wouldn't put economics on that level, but they often seem to gravitate to similar mistakes.
Economics already has huge credibility problems.
I agree with the sentiment but the truth is ballistics is several order of magnitude simpler than biology/biochemistry. I find medicine fascinating, but the more I learn about it the more I am astonished at the huge mismatch between perception of the field and what we actually know. And have always more respect for practitioners.
> Paracetamol was first made in 1877.[21] It is the most commonly used medication for pain and fever in both the United States and Europe.
> How it works is not entirely clear.
What is represented by this pivot towards emperical testing everything is a concession that there are no universal economic laws from which outcomes can be derived, just an ad hoc corpus of facts. I see why they don't want to give up.
However, over time they have hidden variables that no one knows how to measure. Say to day you are in a bad mood. You go to a store pick up an item and go 'nah not going to buy that'. Then tomorrow you are in a good mood. You go to the same store and buy that item. Your neighbor does the same thing but does not buy it at all. Economics is very bad at figuring out what that even meant. But to know if handing people cash, or taxing more/less, or changing policies you kind of need to know what it did mean. Experimenting is a good idea which we have been doing for a long time. But the models still do not match.
The other issue is many of these things are huge systems (macro economics). You can change one little thing and it has an effect on 3 other things that you did not want. To use the classic micro economic model. The pizza joint. I raise my price because (MR=MC). I sell less pizzas but my profit is up. Profit does not come from nowhere. My customers paid more. So they can buy less of something else. I bought less items from my wholesalers so their profit is down. The gov gets more money because of taxes. One little change touched dozens of other stores/governments/people not even related to me. Getting that model right is tricky with thousands of hidden variables that more like functions.
Modern physics uses many of the same methods as softer fields like economics. The problem in economics (and other soft sciences) is the low standard of evidence they're willing to accept.
The editorial Batson is reacting to (by Yao Yang) makes the point that the largest economic development project that lifted the vast majority of people out of poverty over the past 20 years was orchestrated by China, and the RCTs/small scale interventions that Duflo et. al. won the Nobel for had no role or relevance there. He focuses more on the policies guided by the "classic" development economists like Solow which emphasize domestic savings and investment.
Batson's post delves into who in China actually was responsible for the economic policy changes that created that development.
Personally, I welcome the the addition of RCTs to the economic research toolkit. For too long, economics has wrapped itself up in a mathematically complex knot that bears no resemblance to the real world. Behavioral economics has started to crack that by applying common sense, though too often they have dramatically overextrapolated their hard-to-replicate results.
Hopefully economists can see RCT as a tool which can be used where appropriate, rather than an entirely new paradigm that much be applied to everything (as they did with highly mathematical economics).
> The prize, awarded in early October, recognised the laureates’ efforts to use randomised controlled trials (RCTs) to answer social-science questions.
e.g.India with a comparable population to China can only distribute so much with a 2.7 T $ economy compared to China with a 12 T $ economy.
I definitely feel that at least a few Chinese policymakers deserve the Nobel prize for actually lifting people out of poverty. You can debate their methods but i think most people from Western schools of thought come with an inherent bias thinking what worked for them over so many decades would've similarly worked for China which is not really true. This has to do with cultural difference, huge population and many other numerous reasons.
This lack of serious inquiry in the face of enormous empirical evidence that contradicts theory is what makes Economics a dying field.
Big questions are great and I'd be all for focusing on them if we were capable of coming up with reasonable answers. But all we've managed are DSGE models and they're about as useful as reading the lines in goat livers.
I'll take the RCTs thank you and with a side natural experiments. Duflo is the most significant economist of my lifetime. Thanks to her economics has some basis for calling itself a science.
From what I can tell in admittedly fairly brief reading, a single member of the family is trying to remove the name, while the rest of the family simply want it distinguished from the other prizes while not objecting to the use of the name per se.
This statement from Peter Nobel[0], the man actively trying to remove the name, presumably sums up the best arguments he had available against using the name, and makes no mention of other family members trying to deny usage.
[0] https://rwer.wordpress.com/2010/10/22/the-nobel-family-disso...
This is the part that makes it a sham. You call it that to give it a legitimacy it doesn't have. The entire thing is a project in giving economics as a field a legitimacy it doesn't have.
>It is a deceptive utilisation of the institution of the Nobel Prize and what it represents.
https://rwer.wordpress.com/2010/10/22/the-nobel-family-disso...
Not to mention one of its awardees, Hayek.
https://www.nobelprize.org/prizes/economic-sciences/1974/hay...
Which is precisely the problem, and the intended effect. It's a propaganda effort.
But, ultimately, I think the researchers would probably answer with something closer to "I don't care if it's economics. I consider it both interesting and relevant", and so would the Nobel judges.
Unlike any art economics can make predictions, and we can test the validity of such predictions. That our models are insufficient right now doesn't mean we should cast the whole discipline into the toilet.
Agreed.
> we can test the validity of such predictions.
Nope.
> we should cast the whole discipline into the toilet.
Agreed.
I've always been a bit uneasy about economics because people's behavior depends on their beliefs, and their behavior as economic agents especially depends on their beliefs about the the economic theory that motivated the design of their habitat.
I expect that if a population that grew up in a system designed under assumption X and happened to eat well and always have a roof, then they will behave in a way that confirms theories compatible with X.
On the other hand, if they experienced extreme economic strife under leadership that believes in the validity of X, then--as data points--they're more likely to influence economic theory testing in the opposite way.
How can you know whether your predictions are correct because they're objectively true about human behavior, versus them being correct because your sample set has been influenced by the same economic theory that motivated the prediction in the first place?
Is statistics really so powerful that it can eliminate the circularity from the situation?
Edit: Your DVs don't change the facts.
They should have to do that, but they don't. Which is precisely the point, and why they didn't name it something else.
Isn't RCT what the medical world, psychologists and sociologists have been doing for decades?
It was added years later, sponsored by a bank, and is officially called "The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel". It exists purely to legitimize a neoliberal financial ideology and has nothing to do with scientific rigor.
Part of the (harsh) reality in the dislike of economics recently has been that the Behavioural Economics folks (yay Ariely!) have made it a point to start introducing experiment-based rigour and protocols. That part I find fascinating at least.
In the Ebola vaccine trial, they did away with the control groups the moment they realised that the vaccine was effective, sacrificing their nice full statistical analysis for (in this case sensible) common sense. So there are always exceptions to RCTs and there are always cases where they are not the suitable method from the outset.
Is the point of the article that economists are not rigorous enough and that Banerjee, Duflo and Kremer try to be more rigorous, bringing RCTs into economics where they are traditionally more in the biological sciences?
maybe don't thumb suck answers for those questions then as is currently the norm
- ethics. Example: is democracy good for economic growth? Of course one could randomly engineer coups in some countries but that's probably not appropriate.
- cost. Example: how much do people change their labor force participation when taxes change by 1%? where a RCT would be "let's give a _lot_ of money to people" and see what happens.
- situation where it is not appropriate. Example: why did Europe rise to prominence (aka the great divergence)? There is not much to randomize here.
Note that RCTs have shortcomings anyway (see for instance [0]).
The vast majority are certainly not, however, idiocy or ill-intent are not required to fall prey to many common causes of inaccurate results. Smart people trying their best to do good work still frequently succumb to errors and this is especially true in the less 'hard' sciences.
That's why the push for increasing rigor with RCTs and other methods is important and necessary.
"Hard" scientists like to pat themselves on the back for rigor, but they get that because they're studying comparatively simple things. Studying the lives of people is hard, but it's also important. It affects public policy, which in turn affects people's actual lives. That public policy gets created whether it's being studied or not -- the studies are hard, but they're better than guessing, and slowly they can build up a picture that makes them better. It's a bit like medicine: we're not going to stop treating people just because we don't understand the mechanism of action and can't guarantee that it will work.
This breakthrough is about finding ways to use the many villages found in poor countries to even attempt to do an RCT, and to come up with mathematical ways to account for the fact that the trials aren't really randomized. Aid had previously been given based on people's best guesses about what would work, which would maximize the value of the aid given if the guesses were correct, but it's hard to measure if it weren't. Aid has been beset by misguided theories and lack of measurement -- good intentions, but often ineffective.
Yet medicine actually focuses on scientific measurement of effects. They don’t just throw their hands up and go, “experiments that affect people’s lives are too hard.”
Economic or social studies are a lot more nebulous.
In economics, you are studying vast systems. For the majority of questions, it is impossible to isolate some part of the system and control and measure all the inputs and outcomes. That's probably obvious for macroeconomics: You can't have FED raise or lower interest rates based on a random number generator. And even if you could, you would still need a second United States to act as the control group.
It's mostly also true for microeconomics. Consider the difficulty of studying UBI. The largest such studies gave a basic income to a small African village, for a limited time of maybe two years. But the idea, and its opponents, mostly deal with the life choices people make, requiring essentially life-long guarantees. And even just knowing to be part of such a study, or continuing to live in a society that hasn't changed, is likely or at least plausible to change the outcome to render the study meaningless.
The thing is, these complication doesn't explain why nobody overcame them until Kremer, Duflo e. al. started their experiments in the 1990's. Their work appears to be a simple adaption of methods from other fields to studies in developmental economics, not any sort of technological development. (This is one of the earliest papers cited in the motivation provided by Nobel foundation: https://pubs.aeaweb.org/doi/pdfplus/10.1257/app.1.1.112 it does some linear regression at the most)
With creation of new technology ruled out as the blocker for performing the experiment, you are basically left with internal and external sociological explanations.
And text books. "Causality" by Pearl about causal models in general. "Causation, Prediction, and Search" by Spirtes about how to learn the models from data.
For example assume the world consists of three random variables A, B, and C. If A causes B and B causes C (as DAG A -> B -> C), then A and C are correlated. But if the model is A -> B <- C, then A and C are not correlated. But conditioned on B, A and C are correlated in A->B<-C and not correlated in A->B->C. So you can falsify such causal models without an rct
RCTs didn't start with Duflo. (Duflo isn't even the first to win for RCTs -- Kahneman and Smith won in 2002 for experiments.) Experimental economics dates back to the 70s, but it always suffered from the same problem as psychology -- most experiments were conducted on students, and the interventions were always small-scale.
RCTs in development economics are much bigger scale because there are rich NGOs willing to spend big money on measuring the efficacy of interventions, and willing to work with economists to do it. This is not without controversy. A development RCT involves an economist from a rich country flying to a poor country, and then running an experiment on the inhabitants of that country. Not everyone thinks that's okay.
The RCTs also rely on the fact that economists come from coun
It's harder to know things we can't do experiments, but we can still know them. In economics, there is a rich tradition of relying on "natural experiments", which is where something like a natural disaster or a law change allows researchers to examine the effects. This is how it was shown that the effect of minimum wage increases on employment is very small. The financial crisis falsified an entire school of macroeconomics.
Fields such as mathematics and philosophy were also around in Nobel's time and he didn't think they were worthy of a prize either. The difference is those fields aren't associated with an organization that literally prints money to buy their way in.
I doubt you have evidence of this, beyond the simple fact that he didn't personally establish a sixth prize. If such evidence existed, I think it's unlikely that the Nobel Foundation would've agreed to administer the prize in the first place.
That said, at this point, I'm not sure what the difference would be anyway. The Nobel prizes (including the memorial one) have become a globally admired celebration of human achievement, the personal beliefs and shortcomings of the 19th century arms dealer who established the prizes notwithstanding.
If the key properties are “Signals the same sort of contribution to human understanding. Administered by the same org as admins the other Nobel prizes. Has the same credibility.” Then you are merely technically correct.
They seem to be perennially awarded to academic staff at the university of Chicago for coming up for a new way of pricing something or interpreting the markets. Nothing earth shattering or outside the realm of orthodox economics (beyond a few exceptions such as game theory).
No such thing exists.