That is the title of a recent book by Clayton Christensen, an admired management guru. It’s a question (in slightly more modest form) I frequently ask. I asked it when I was at the Telecom Regulatory Commission of Sri Lanka: how could we measure what good we did? Result: Telecom Policy and Regulatory Environment tool. We ask it all the time at LIRNEasia.
Establishing causal relationships is tough in social science, a field defined by open systems and knowledgeable agents. It is extremely difficult with regard to policy actions and outcomes. In 1998-99 when I was the public face of telecom reform in Sri Lanka, I used to get lots of invitations to speak at business events. In these talks I had to maintain a delicate balance, between giving the then government and its predecessor the right amount of credit for the incredible outcomes that were becoming visible. The government in power had been in place for four years and had partially privatized the incumbent operator and given its management to a Japanese investor, among other things.
Yesterday, I was pleased to have the opportunity to share our thinking on measuring the efficacy of our research and capacity-building work with colleagues from RMIT, Swinburne and ACMA at a seminar organized at RMIT. It was intriguing to hear that some of the participants thought that Bangladesh gave more room for genuine policy inputs from those outside government than Australia. I know first hand the limitations of the policy process in what I like to call my countries, but they do not. Did not get much help in solving the puzzle of measuring the efficacy of CPRsouth. Here the surprise was that Australia seemed to lack a forum such as CPRsouth for two way interactions between policy people and scholars.
Yesterday I heard a speaker at the ICTD 2012 conference in Atlanta say that 80% of the USD 4.2 billion spent by the World Bank on ICTs had been labelled as a failure by the Independent Evaluation Group. I had read the study in detail, and had blogged about it. I still wonder how language such as that below, taken from the report, can be interpreted thus: In other priority areas, including ICT applications, the Bank Group’s contributions have been limited. Targeted efforts to increase access beyond what was commercially viable have been largely unsuccessful.
It’s always a challenge to decipher the special language of evaluation reports, but this para in the recent evaluation report on the ICT activities of the World Bank does seem like an indictment of universal service programs. 4.28 Equity and integration of marginalized groups have been more effectively supported by Bank support for ICT policy and sector reform than by operations specifically designed to achieve these goals. ICT operations that supported reforms to introduce competition into the sector, when successful in supporting those reforms, have had significant impact, especially in access to cellular telephony services. This increase in overall access has had a spill-over effect of providing access to the underserved.
As a results-oriented organization, that is a question LIRNEasia has always been interested in. The discipline that seeks to answer that question is evaluation. They recently held a conference in Sri Lanka. We are ratcheting up our emphasis on evaluation now that we have a substantial body of work to talk about. A key element in this will be Chanuka Wattegama’s participation in the most important evaluation training program currently being offered, the International Program for Development Evaluation Training offered every Summer at Carleton University in Ottawa, with the cooperation of the World Bank and IDRC.