The IDRC Asia Office was kind enough to permit publication of a part of a concept paper I wrote for them. Here is an excerpt: At present, the principal methods for understanding users or demand are quantitative research (representative-sample surveys) and qualitative research. The former is used primarily for understanding “what” questions and the latter for “why” questions. Quantitative research is very costly. Its limitations include problems of recall and different forms of bias.
As part of electricity work LIRNEasia has made recommendations on the importance of DSM in Sri Lanka. Effective DSM is not possible without smart meters and that was an important part of the message, when we were invited by the Colombo Electricity Board (CEB) to share our research with their senior management. So it was with great interest that I perused the research of one of the winning finalists from a Big Data Challenge conducted by Telecom Italia (and partners) with data pertaining to the territories of Milan and of the Autonomous Province of Trento in Italy. The datasets covered telecommunications, energy, weather, public and private transport, social networks and events. The researchers utilized smart meter data and behavioral data extracted from the Telecom Italia’s transaction generated data to predict peak daily energy consumption and also the average daily energy consumption for each line through the electrical grid of the Trentino Province.
The work that we have been doing using mobile network big data over the last year, has been challenging on many fronts. I’ve spent some time reflecting on some of the analytical challenges that are faced in the Big Data paradigm and the common fallacies that I sometimes find in the broader discussion I see on the subject. What is below are some of my preliminary thoughts, which I am working up into a paper. Still a work in progress. Comments welcome.
Today was the first public airing of our big data for development research results. It was a small amount of time, so we focused on a limited set of issues. So we showed that anonymized data sets can easily substitute for costly traffic studies. Slides.
Issue of discrimination coming up in big data policy review. The value of big data is in understanding the consumer. But with understanding comes the ability to discriminate. Not all discrimination is bad. But some may be.
The recent kerfuffle about Google flu trends showed all kinds of critics of big data come out of the woodwork. This is normal for anything new. I am sure there was an outpouring to hostility to the motor car when the first accident occurred. What I found useful was the cooling of the hype associated with big data. I have no doubt it is big and will, in fact, lead to a data economy.
This is continuation of discussion with Sunil Abraham and Steve Song. It got a little too long for a comment. The problems under discussion are difficult. So it’s good that we have an active discussion. We could have a discussion about all sorts of approaches to privacy.
US agriculture was early to use ICTs to improve efficiency. I recall sharing stories of information-savvy farmers with my classes in Ohio in the early 1990s. Now data is available of soil and weather conditions at a micro-level and farmers are beginning to be concerned that this big data when combined with other data could result in the rigging of futures markets: And the interested parties are familiar names on the farm—names like DuPont and, of course, Monsanto, which is on a buying spree. Monsanto bought the high-tech farm equipment maker Precision Planting in 2012. Last October, it bought the Climate Corporation, a data-analytics firm that provides weather-related farm services and crop insurance, and is also handling Monsanto’s fledgling data-related services.
The Financial Times carried a good discussion on the failure of the Google Flu Trends model to predict. High quality global journalism requires investment. Please share this article with others using the link below, do not cut & paste the article. See our Ts&Cs and Copyright Policy for more detail. Email ftsales.
Yesterday I listened sporadically to a live streamed conference on Big Data. Sporadic was not intentional. I am in Dili, Timor Leste, where most connectivity is via satellite with latencies in the 700ms range. Anyway, the focus was not on big data per se. They talked about all sorts of things, mostly open data (in the parts I heard) and crowd-sourced data.
As part of our big data work, I have been looking at the 2012 Sri Lanka census preliminary results based on five percent of the responses. The picture that emerges is a far cry from the slums described in this Economist summary of research on slums: Yet the MIT paper, which offers simple statistics about 138,000 slum households from around the world, suggests that slums are often an impediment to advancement. Poor hygiene, and the debilitating illnesses it propagates, is one curse. The majority of slum-dwellers in the MIT sample have no private latrine; in one Mumbai slum, taps are shared by more than 100 people. According to the African Population and Health Research Centre hygiene is regularly worse in slums than in rural areas.
Since 2006, when the majority of the world’s population became city-dwellers (can’t use the original term “citizens” because it has now lost its connection to cities), there has been a great deal of interest in understanding these engines of economic growth. Here are some findings from Spain, in the process of being replicated in Asia. The results reveal some fascinating patterns in city structure. For a start, every city undergoes a kind of respiration in which people converge into the center and then withdraw on a daily basis, almost like breathing. And this happens in all cities.
Prof Hal Abelson of MIT recently shared his thoughts on privacy in the digital realm, at a online alumni webcast. Amongst the noise that one hears on this topic these days, his thoughtful comments resonated. Partly for sharing and partly for my own memory, I felt it justified a blog post and I capture his main points below: People don’t really know what they want when they think of privacy. They describe their privacy needs through use-case scenarios for e.g.
A new robot equipped with multiple sensors that can collect information from its surroundings that can be matched against “big data” streams has been announced. The facts are interesting. Even more intriguing is the allusion to Minority Report. K5 also raises questions about mass surveillance, which has already set off intense debate in the United States and Europe with the expansion of closed-circuit television systems on city streets and elsewhere. The Knightscope founders, however, have a radically different notion, which involves crime prediction, or “precog” — a theme of the movie “Minority Report.
Little data is as bad a term as big data. Really tells you very little. But sadly that is what the New York Times has chosen to use. And I have not had time to come up with something little more insightful. David Soloff is recruiting an army of “hyperdata” collectors.
I was reminded of that old chestnut about a flagman having to walk in front of early automobiles when I heard some participants talk at the workshop on big data, social good and privacy. Imagine imposing inform and consent rules on transaction-generated data (big data) belonging to large corporate entities such as mobile operators. They need the data on user mobility patterns to manage their networks; they need financial transaction data to manage their finances. All these things can be covered under broad inform and consent procedures that will be presented to customers as they sign up. What will not be possible would be to permit use by third parties for traffic management, energy management, urban planning etc, since these uses could not be conceptualized at the time of signing up customers.