Based on feedback received from participants at CPRsouth in Manila and the WDR Expert Forum in Singapore on the Telecom Regulatory Environment (TRE) assessment conducted in six countries in South and South-East Asia in 2006 (India, Indonesia, Pakistan, Philippines, Sri Lanka and Thailand) we have decided to refine the methodology.

Dimuthu Ratnadiwakara and Shamistra Soysa, new additions to LIRNEasia‘s team in Colombo present the preliminary results from the adjusted methodology.

The original method categories had different weightings across countries. For example if category A had more responses than category B, the results were then skewed towards category A. To remove this bias and ensure each category in each of the countries had an equal percentage of input, the total number of response categories was reduced from five to four, because India had 0% responses from the ‘former regulatory staff’ category. Thailand was also removed from the analysis because the sample sizes for each category were significantly smaller than the sample sizes of the other countries.

The suggested approach re-analyzed the original responses to the TRE scorecard using a normalization method. Initially, the number of responses were added together from each category and then the scores in each category were divided by the total number of responses and multiplied by 25% of the total number of responses. This would ensure that each category would have 25% input to the final scores.

However, this method did not show significant differences between the original scores and the normalized scores. This was explained by the small number of category 2 (financial institutions/investors) responses, which increased the weightings.

Therefore the two recommendations to improve the reliability of the TRE results:

1. Re-think categories (especially category 2) because researchers have had trouble getting responses in that category, so when you try to replicate the study in places like Maldives, Bhutan, Afghanistan, it will be even harder.

Category 2 could be merged with either category 1 (industry) or category 3 (consultants, lawyers, etc), although this may not fit with category 1 because incentives are not exactly the same. But both category 1 and category 3 are current and future investors.

There is a need to consider whether the groups have the public or private interest in mind when grouping them into categories. For example, in favor of market entry or not? Need a balance between the ‘corporate’ types as well as the public-interest-types.

Issue: Why are donors placed with investors and finance houses (category 2)?
This is only suitable for donors like IFC (international Finance Corporation) because their objectives are more in line with commercial interests. Others should be placed with the more public-interest category(s) – 3 & 4.

Possibility of including a ‘Dummy variable’ of sorts for small countries – e.g Bhutan & Afghanistan- so that the results do not get skewed, due to lack of responses in certain categories.

2. It is suggested that at least and up to 10 respondents for each category are obtained in order to obtain more reliable estimates and even allow for comparison between categories. Why the number 10? Because it would allow non-parametric testing. Three categories will also make things simpler.

Another method could be to take average scores of each of the 3 categories (unweighted), and take the simple average of all the categories (assuming reliable estimate for each category).

But this poses the problem of ‘reading backwards’ and working out the disaggregated results, because we will have to disclose the individual categories averages. For example, if a regulator realizes that operators are criticizing them, especially in countries like Maldives where the number of operators is only 2, the regulator may give the operators a hard time.

There is a plan to make a manual which will allow other researchers to read about the TRE and implement it on their own, so the final method has to be as robust as possible.