Can development policy and human behaviour be measured in the same way that medicines are tested? This question lies at the heart of the work by the 2019 winners of the Nobel Prize in Economics, Abhijit Banerjee, Esther Duflo and Michael Kremer. It is also the reason their approach to economics has received much criticism over the years.
“The research conducted by this year’s Laureates has considerably improved our ability to fight global poverty,” said the citation by the Nobel committee. “In just two decades, their new experiment-based approach has transformed development economics, which is now a flourishing field of research. Their experimental research methods now entirely dominate development economics.”
The committee even put together a “popular science backgrounder” as a primer for those seeking to understand what was novel about the approach used by Banerjee and Duflo, who also happen to be married to each other and frequently conduct their research together, and Kremer.
These new research methods are not without controversy (which has nothing to do with Banerjee’s involvement with student politics at the Jawaharlal Nehru University or his assistance to the Congress).
But before we get into that, first an explanation:
What are these ‘experimental research methods’?
At the crux of the approach are Randomised Controlled Trials, or RCTs for short. In the medical research industry, for example, RCTs are the accepted way of gauging the efficacy of new medicines.
In an RCT, there are two groups of people: The experimental (or intervention) group, to whom the new medicine is administered, and the control group, a similar set of people who are not provided intervention, or given a placebo. Crucially, once a population is picked, the individuals falling into either group are supposed to be random – to prevent distorted results.
The success of the medicine is tested by measuring the difference between the average indicators in the experimental group to the average indicators of the control group. If there has been a significant change, such as the medicine working to alleviate an illness in the experimental group for example, the intervention is considered successful.
The 2019 Nobel laureates were pioneers in expanding this approach into the field of developmental economics. In place of medicinal intervention, this approach seeks to alter policy design, introducing changes to the lives of those in the experimental group to see if different policy can have better outcomes.
The committee’s explainer gives an example from schools in low-income countries:
“One way of boosting the teachers’ motivation was to employ them on short-term contracts that could be extended if they had good results. Duflo, Kremer et al compared the effects of employing teachers on these terms with lowering the pupil-teacher ratio by having fewer pupils per permanently employed teacher. They found that pupils who had teachers on short-term contracts had significantly better test results, but that having fewer pupils per permanently employed teacher had no significant effects.”
Why is this approach considered unique?
In the views of many, some of the older approaches to developmental policy have depended too much on qualitative assessments or policies that are designed around huge data sets and inferences about how people act rather than trials that produce evidence and numbers of actual behaviour. The RCT method pioneered by the three laureates breaks the question down into smaller variable and attempts to measure outcomes in a manner that can help direct policy.
“The Laureates’ research findings – and those of the researchers following in their footsteps – have dramatically improved our ability to fight poverty in practice,” the Committee said in its briefing. “As a direct result of one of their studies, more than five million Indian children have benefitted from effective programmes of remedial tutoring in schools. Another example is the heavy subsidies for preventive healthcare that have been introduced in many countries.”
Indeed, so popular has this method been that it has nearly taken over the developmental economics field. In a paper in 2017, it was referred to as “the new gold standard”, with a significant amount of research in the space relying on the methodology in some form.
“The effect of such rigour on policy analysis is considerable,” wrote Mihir Sharma in the Business Standard. “For countries like India where resources, particularly in the least developed states and areas, are hard to come by, the results of RCTs are vital input into decisions. They ensure that vital years and budgets are not wasted.”
Why is it controversial?
Despite, or maybe because of its popularity in the field, the experimental approach has also received much criticism.
At the base level, some have argued that an over-reliance on RCT-based research turns questions of poverty into bite-sized problems, with a focus on correcting individual behaviour, instead of grappling with the larger, systemic issues at hand.
“The real problem with the ‘aid effectiveness’ craze is that it narrows our focus down to micro-interventions at a local level that yield results that can be observed in the short term,” wrote 15 leading economists including three Nobel winners, in a piece in the Guardian in 2018. “At first glance this approach might seem reasonable and even beguiling. But it tends to ignore the broader macroeconomic, political and institutional drivers of impoverishment and underdevelopment.”
Although some have dismissed RCTs entirely, for many researchers in the field one of the real problems with the approach is what can only be described as the moral superiority it seems to command. Since it was popularised, the experimental approach has been sold to the field and the public as a sort of perfect, unbiased, evidence-based process that can provide hard numbers and proof of successful policy interventions.
But humans are extremely complex, and narrowly designed experiments – particularly those that rely on a “control” group without fully understanding all the variables and imperatives that drive behaviour – may not tell us very much.
As economists Farwa Sial and Carolina Alves write:
“Poverty alleviation, however, is a hugely complex subject that touches on the strengthening of institutions, the health of governance, the structure and dynamics of markets, the workings of social classes, macroeconomic policies, distribution, international integration and many other issues, none of which can be replicated from one context to another. That means that analyses of poverty have to be based on a critical examination of processes and actors that cannot be ‘controlled’ against – thus violating the principle of RCTs.”
Another criticism frequently raised is that the trials at best tell us only about the populations being examined, and cannot be extrapolated. As Angus Deaton, another Nobel laureate who works in the field of development economics, explains, “demonstrating that a treatment works in one situation is exceedingly weak evidence that it will work in the same way elsewhere.”
Researchers using RCT tend to pride themselves on not making presumptions about the populations they are dealing with, yet, in Deaton’s reading, it is this approach that makes it impossible to transport the results to any other population.
More critiques of the method can be found here and here.
Meanwhile, the award to Banerjee has also led to a very different kind of praise and debate – are Bengalis the pre-eminent Nobel-winning community of South Asia?
Limited-time offer: Big stories, small price. Keep independent media alive. Become a Scroll member today!
Our journalism is for everyone. But you can get special privileges by buying an annual Scroll Membership. Sign up today!