A graduate school friend of mine recently sent me an article about a new public opinion research tool based on the "complex" analysis of social media content -- primarily Facebook and Twitter posts. For lack of a better name, let's call it Social Media Monitoring. Of course, as someone who has spent a career trying to understand the dynamics of public opinion, I tend to discount non-sampling-based opinion research methods. Nonetheless, I am not entirely against using non-sampling-based methods to measure public opinion. As anyone who has read my posts on this blogsite knows, I am a strong advocate for prediction markets which provide real-time measures of public opinion but are not sample-based.
So, I do not dismiss the notion that quantifying Facebook and Twitter posts has merit. After all, there is real evidence that changes in social media content relate to real world behaviors. For example, Google researchers recently demonstrated the relationship between queries on Google's search engine and economic activity (Predicting the present with Google Trends ). A rise in Google searches for Ford cars corresponds to a rise in Ford sales. But, even by their own admission, inquiries on Google are merely concurrent indicators of economic activity and not necessarily a good forecasting tool.
But the article sent to me about this new social media monitoring company does concern me because the company appears to confuse behavior with opinion. The company mentioned in the article, Crimson Hexagon, mines social media posts and tweets to develop estimates of public opinion. (Harvard developed analytics tool measures public opinion ). Crimson Hexagon offers a research product to their clients based on an analysis of these online conversations. Nothing wrong yet. It is certainly reasonable that Microsoft or Apple would want to know the direction and tenor of online conversations concerning their products. More importantly, I do not doubt Crimson Hexagon has developed a valid methodology for quantifiying the unstructured data generated by Facebook, Twitter, etc. But early in the article the Crimson Hexagon CEO Scott Cenurino describes his company's research product as the measurement of public opinion from unsolicited data. Yow! Now we have a problem. Good public opinion research -- and research in general -- requires at least an attempt by the researcher to maintain control over all factors that may affect the result of study. This is particularly important when you are engaged in research where people are the subjects. By abdicating that requirement, Crimson Hexagon's research results will be prone to the selection bias problem. In fact, I would say its guaranteed.
[Go to this link to see a presentation I gave at the 2010 Intelligence Community Assessment Conference on how selection bias can make you believe cats can fly: The Science of Falling Cats ]
We already have evidence that the Pentagon manipulates social media content for its purposes (-Pentagon Seeks to Manipulate Social Media for Propaganda Purposes ). It would be short-sighted of the Pentagon not to be engaged in such an activity. And the number of U.S. companies devoting significant effort into creating their own social media presence is too numerous to list here. But therein lies the problem. Social media content is subject to active manipulation. Blogging and tweeting is a behavior, not an attitude. Therefore...social media monitoring is not measuring opinion, it is measuring behavior. In other words, one cannot know for certain that Crimson Hexagon's social media measurements reflect true opinion within even the Facebook-blogo-tweet-esphere, much less the general public. They appeal to the natural bias within most of us to view data as more valid than no data. Crimson Hexagon crunches millions of data points! That's worth something, right? No, not necessarily. Selection bias does not necessarily go away as you collect more and more data. I often accuse market researchers of being modern-day witch-doctors when they compromise scientific standards and I'm afraid that is what Crimson Hexagon is engaging in when they promote their social media measurements as valid estimates of public opinion. These measurements are not.
"There you go again, pissing on someone else's good idea because you didn't think of it first." I admit, that is part of it. And I am not saying that measuring social media content doesn't make for a great independent variable -- in other words, as a variable that one can use to explain variation in another variable of interest (i.e., the dependent variable). After all, the social media clearly impacts the national dialogue on many issues, if only because news organizations are constantly quoting social media in their news stories. But public opinion researchers must not cede ground to Crimson Hexagon or Google Analytics when they make claims they can measure public opinion. They are measuring something -- perhaps something important -- it just isn't public opinion.
No comments:
Post a Comment