In an overview of studies on India in the United States, Devesh Kapur of the University of Pennsylvania has some less than complimentary things to say about RCTs. They mirror some of my comments about systematic reviews here, the next layer of RCTs, though I do not say anything about the benefits to reseachers like Devesh does.
By contrast, there has been a considerable increase in India-related work in the social sciences. The field has become much more empirical and India offers several advantages for a researcher: large sample sizes, heterogeneity in multiple dimensions, relatively low cost of gathering data, and weak official oversight (which, in any case, is unlikely to be enforced). It would be hard to do many of these trials in the US or China. As a result, custom-designed household surveys and randomized control trials have proliferated, most supported by external funding, which have generated new understandings about the country. But what has been the policy impact of this work?
Suffice it to say the effects have been much more positive for the careers of US-based researchers than for India. Ironically, some of the very strengths, such as the stress on identification and causal inference, have been a source of weakness. The stress on these methods as “the gold standard” comes at the cost of relevance and timeliness. Only certain types of questions can be addressed by these methodologies. This is not to say there aren’t excellent studies that address important policy questions. But more often than not, even if they can address them, the costs and duration of these studies means they are more useful as citations than policy.
When asked how many of these expensive RCTs had moved the policy needle in India, Arvind Subramanian, Chief Economic Advisor, GOI, was hard pressed to find a single one that had been helpful to him in addressing the dozens of pressing policy questions that came across his table. By contrast, the compiling of just some key facts on learning outcomes by Indian NGO, Pratham, has had a big impact on policy discussions in education, because it is backed by a degree of specific knowledge and engagement that is more credible and persuasive. One could question whether “relevance” or “timeliness” are a valid standard for good research—yes they are, when those are precisely the reasons given to funders for these projects.
Comments are closed.