Politico-Facebook Sentiment Analysis Will Generate "Bogus" Results, Expert Says
BY Micah L. Sifry | Friday, January 13 2012
Thursday morning, Politico announced that it was joining with Facebook to "measure GOP candidate buzz" and give its readers an "exclusive look at the conversation taking place on the social networking site" ahead of the January 21 South Carolina primary. "Mitt, Paul winning Facebook primary" was the headline on their first story on the project.
"'Social media has forever changed the way candidates campaign for the presidency,' said John F. Harris, editor-in-chief of Politico, in a press release about the new partnership. He added, “Facebook has been instrumental in expanding the political dialogue among voters and we couldn't be more excited about the opportunity to offer our readers a look inside this very telling conversation.”
At the core of the partnership are two projects. One is completely uncontroversial: Politico's editorial team is going to survey voting-age Facebook users on a daily basis and report the results each day. Presumably these will be randomized according to typical polling methodologies. The other one is completely "bogus," as one expert put it to me: Facebook is analyzing its users' status updates, postings and comments that refer to the candidates, and assigning positive and negative values to them, producing a daily track of their supposed ups and downs.
It's called "sentiment analysis." It's the heart of the pretty charts and graphs that the two companies rolled out to tout their partnership. And it's total bunk.
Here's the issue: Counting the number of times a candidate's name is mentioned on social media and noting what words appear alongside those mentions can illuminate broad trends. You can report that "more people talked about Candidate X today" and "Y percent of that group used word ZZZZ in their comment." But you can't make any kind of meaningful judgment about what those people intended by that usage without asking them.
Someone who writes "I'm so happy that Newt Gingrich is staying in the race" might be a genuine Gingrich fan, or they might be someone who hates him, but likes that he's staying in the race because he's entertaining, or because they think he's hurting the Republican field. But "sentiment analysis" is still such an embryonic field that serious researchers tend to avoid any hard claims about whether such a statement is positive, negative or neither.
Facebook says the tool it is using, Linguistic Inquiry and Word Count, is a "well-validated software tool used frequently in social psychological research to identify positive and negative emotion in text." It is an often-used tool: It can be useful, researchers said, to find frequent words and how they are associated. But several researchers I spoke to were highly skeptical about making definitive claims about what the word analysis might show.
Marc A. Smith, director of the Social Media Research Foundation, told me, "I share your skepticism re: 'sentiment' analysis. Irony is a tough nut to crack."
"That said, I do put great faith in simple text volume tracking," he added. "When the names of candidates rise and fall in volume that is news. When the words that co-occur with them change, that is news as well."
"So my preferred approach is simple frequency measures on keywords and *word pairs* rather than generating percentage measures of positive or negative sentiment."
Smith sent me a quick analysis that he did of Twitter comments in the last 24 hours from people following Mitt Romney. The top word pairs he found were:
mitt - romney
romney - heads
auto - industry
south - carolina
romney - came
ron - paul
victorious - romney
far - worse
bain - bad
attacking - romney
Here's the full "keyword network graph" that Smith developed (using NodeXL) from the same data set. Words are linked because they occurred next to one another in a tweet, filtered so all words with only one connection are removed.
As you can see, this isn't something you can write a headline around. Serious efforts to analyze online conversation en masse produce much less satisfying results than Politico's dubious claim to be tracking who is winning "the Facebook primary." (And even Politico's story Thursday on the "Facebook primary" admits that the data they're collecting often don't make sense on their own terms.
As reporter Rachel Van Dongen wrote, "Declaring a winner of the Facebook primary — or the candidate viewed most positively over the past month — is far more difficult than tallying votes. Santorum didn’t see a spike in positive postings on Facebook around his Iowa win, but Gingrich did."
Say what? How could Gingrich's cranky post-caucus speech on live TV have generated positive buzz for him?
Libby Hemphill is another expert who shies away from making claims about analyzing sentiment in social media. An assistant professor of communication and information studies at the Illinois Institute of Technology, she is working on a study of language use among elected officials.
"Analyzing text from social media is especially hard because it has so little context and is so short," she told me. "I think most of those sentiment reports are bogus for the reasons you mention and more.
"We're working on algorithms for automatically detecting content and manner, but we're avoiding sentiment," she continued. "These sentiment reports make the analysis sound so much easier than it actually is, and I worry that they're giving automated detection and classification a bad rep before we even get off the ground."
Jahna Otterbacher, a language expert who works with Hemphill, concurred. "Even in the case of words that, individually, are completely uncontroversial with respect to the sentiment they convey (e.g., good, bad, etc.) in context, things are much trickier, even before we get to talking about irony and sarcasm (e.g., 'It's so good to see him go' is clearly not conveying positive sentiment toward the person in question.)"
And online chatter, she added, makes analysis even harder.
"With the more casual nature of language that's used in social media contexts," Otterbacher said, "things are even more complicated (e.g., 'He's got it bad for her.')"
I raised these concerns with spokespeople for Facebook and Politico. A Facebook spokesman did not comment for the record. Politico's press team did not respond to a request for comment.
Remember, there are reasonable ways to explore social media metrics around the presidential campaign. Take for example this post by Gavin Sheridan on Storyful. They collected the 53,000 tweets from within the U.S. mentioning a presidential candidate in the 24 hours before the New Hampshire polls closed, and they also focused on the tweets that were specifically from the state. Lo and behold, they found that the volume of mentions correlated to the order of the candidates' standing in the vote: Romney, Paul, Huntsman, Santorum, Gingrich and Perry. But did Storyful run breathless headline claiming that Twitter "predicted" the election (as did Mashable)? No, it just noted that "The New Hampshire data was interesting because it appears to closely match, probably coincidentally, the actual result of the primary."
Or take this post from Google Politics & Elections, which looks at trends in search interest in the different candidates. Santorum, Romney and Paul are all bunched together with a recent uptick in interest shown in South Carolina in their names. Interestingly, Gingrich is trailing those three, and searches for Rick Perry are completely flatlining. There's no rush to claim that this means someone is "winning the Google primary," and instead Google's team just asks readers, "What do you think of these early trends?"
It's not too late for Politico to back off its claim to be mining Facebook chatter for deep meaning, and just stick to simple statements about who's being talked about. But I doubt they'll do that. Whether it's CNN with its fancy and expensive computer graphics (remember the hologram?) or the various debate partnerships between media organizations and tech companies, a big chunk of the political press seems deep in a mindless love affair with technology and social media. After years of scorning the web, now some of them are rushing in the opposite direction, chumming up with companies they should be reporting critically on, and claiming to find meaning in things that are nowhere as clear as they seem.