The work that we have been doing using mobile network big data over the last year, has been challenging on many fronts. I’ve spent some time reflecting on some of the analytical challenges that are faced in the Big Data paradigm and the common fallacies that I sometimes find in the broader discussion I see on the subject. What is below are some of my preliminary thoughts, which I am working up into a paper. Still a work in progress. Comments welcome.
Causation is a central concern of science. In closed systems such as those found in Chemistry and Physics, this is generally not too difficult. In open systems such those that we work on (i.e., telecom use) it is a tremendously difficult problem because multiple factors are at work at the same time and interacting with each other and with the phenomenon we’re trying to establish the cause for.