Establishing causal relationships is tough in social science, a field defined by open systems and knowledgeable agents. It is extremely difficult with regard to policy actions and outcomes. In 1998-99 when I was the public face of telecom reform in Sri Lanka, I used to get lots of invitations to speak at business events. In these talks I had to maintain a delicate balance, between giving the then government and its predecessor the right amount of credit for the incredible outcomes that were becoming visible. The government in power had been in place for four years and had partially privatized the incumbent operator and given its management to a Japanese investor, among other things. They definitely deserved credit. Yet the Act that removed barriers to entry and the main work on licensing had been done by the previous government. They too deserved a part of the credit.

When the policy actions that are claimed to be responsible for a specific outcome happen at the same time, unlike in the above example, the problem gets more complicated. Dueling narratives on the ongoing recovery in Ohio (where we lived and worked from 1987 to 2000) are a case in point:

These competing theories of progress in Ohio, waged 250 miles apart at the same moment, illustrate what may be the most important and confounding dynamic in this election. While most of the debate nationally still revolves around why the economy remains so pathetic, there are several pivotal states — Ohio, Florida, Nevada, Virginia — where things are slowly turning around. In these states, the real issue may not be who deserves blame for economic ruin but rather who deserves credit for a rebound, and what really causes jobs to come back after they’ve been lost. Republican governors are saying that unemployment rates have plummeted because of their pro-business policies. The president is saying that the hard decisions he made earlier in his term are finally starting to pay off. And then there’s Mitt Romney, a congenital optimist who finds himself in the uncomfortable position of having to be a total downer, arguing that there really isn’t a recovery at all. “Trust Me: You’re Still Miserable”could be Romney’s bumper sticker in Ohio.

Matt Bai, one of the best political writers around, does a great job in untangling the debate. But that is because he’s good.

Evaluators who do not understand the inherent complexity of the policy process, and the layers added by the research-policy nexus, rarely nail causality. What Obama did (and when) and what Kasich did (and when) are matters of public record. When LIRNEasia communicates its research to the Indian government (as was the case with our universal service and Access Deficit Charge work), it is not. By definition, other observers in India would not be able to testify to what we did. Yet, a naive evaluator took what Indian observers said at face value and gave no weight to the evidence we placed on the table. That was the equivalent of disregarding Geithner altogether and taking Romney as an unbiased observer.

That’s all water under the bridge. But causality matters, in both politics and in evaluation. The more we understand the difficulties of establishing it the better we will be at avoiding error.