For Measuring Impact of Journalism and Advocacy, Data is Not Just Data
BY Miranda Neubauer | Tuesday, June 3 2014
For Upworthy, often the most important data is no data at all. In a Friday talk at the Columbia Journalism School, Daniel Mintz, director of Business Intelligence at Upworthy, explained that in many instances Upworthy does not make raw data about the success of its posts available to its curators.
"If the data doesn't help us make better decisions, that's not useful data, that clouds judgment," he said. "We get into fights with staff, 'don't take our data away', we hide that data, because it's just distracting."
Instead, Upworthy often replaced the data information with terms like "bestish" to offer insight to its curators on the reception of their content, or photos of Mintz with a thumbs-up or thumbs-down.
That is part of Upworthy's approach to using metrics to achieve its goal to "drawing massive attention to the things that really matter," not just a massive number of clicks. "With clicks you can just keep clicking, " he said, adding that the "zero sum game" is for attention to topics such as climate change, body image, media portrayal of women and bullying.
Going beyond simply counting clicks to using data to inform journalistic or policy goals was a recurring theme among the panelists participating in the first research conference sponsored by the Journalism School's Tow Center for Digital Journalism. "If we're looking at the homepage, we're not interested in [figuring out] the number one story," said Tony Haile, CEO of Chartbeat. "We're considering what would a story in that position normally do, what is the opportunity cost of putting something else there."
Data can give insight on whether a reader stops after paragraph two and does not finish the story, Haile said. "It's not about ranking stories, but how well am I communicating that story ... but considering the story I wrote, what I hoped the response would be, can we adapt the writing to get that response," he said, suggesting making changes to the headline or lede. Data can help answer the question, "is what we thought as important the same as what the audience thought was important." As an example, he cited an instance when the Boston Globe site highlighted a piece about the zodiac signs being incorrect, even while more of the audience engaging with a story about a local theater burning down.
In order for editors to use data to inform how to best package or present a story at the current moment, and "being able to democratize decision making, you have to put it in the hands of people in the right way ... if you just give numbers to people, that's no good," he said, emphasizing that rather it is key to communicate "the core thing users need to know" about how they should act or focus their activities. "
Mintz explained how Upworthy uses analytics to determine whether a video is still playing in a user's browser, including data on how much time all users spent watching. He noted that existing tools such as Google Analytics are often imperfect at measuring the kind of attention and engagement news sites seek to measure. "We noticed at one point that our average time on page [in Google Analytics] was about 20 minutes; we're really good, we're not that good."
But even for all the data available, the panelists agreed that the ability to measure impact is still elusive. "We're not anywhere close to being able to understand importance or true impact in terms of change," Haile said.
Mintz concurred that measuring impact is difficult. "A story that 200 people saw about some corrupt public official, but that got him to resign, which would not have happened otherwise, that's really important and impactful, but how do we capture that?," he asked. With a link to a Kickstarter project, real-world impact may come in feedback from the project owner that links from Upworthy raised $100,000, he said, but often it is more anecdotal feedback from someone who says an Upworthy post changed how they think about a subject. One of the strategies Upworthy has been using has been trying to measure impact of posts in a qualitative rather than quantitative way, he explained, in which an editor might tag a piece as very important in a database, allowing comparisons of the reception of the very important pieces versus the more generally important ones.
"We know a lot about what happens in people's fingers, we know nothing about what happens in their head," agreed James Robinson, director of news analytics at the New York Times.
On a later panel, participants explored ways that data and datajournalism may have an inherent impact on public opinion, how to reconcile the different ways that data and values can inform policy debates and how the emphasis on data requires changes in traditional educational curricula.
"Data is never just data, ... there's a politics attached to it," Tow Senior Staff Associate Jonathan Stray said in an introduction. Pointing to government data releases summarizing the positive economic impact of gay marriage, he asked whether one can "really use data to decide whether two people of the same sex should be allowed to marry? Is that even a question you can ask with data?" He pointed to a recent piece by Leon Wieseltier that criticized data journalism by arguing that many of the important question up for debate were not empirical questions but questions of value. Stray argued that "when we collect data about the effects of gay marriage to try to convince people..., we're not trying to establish some fundamental truth about the world, we're not being scientists, we're trying to convince people that what we believe is right. Our political system would not work without this type of convincing each other."
Expanding on that theme, New York Times Graphics Editor Amanda Cox highlighted a 2012 New York Times tool that let readers examine the monthly jobs report from a Republican or Democratic perspective. "These are all true statements. You don't even have to lie," she said.
Pointing to recent reporting on the VA hospital scandal that led to Eric Shinseki's resignation, she also emphasized that "when you measure the data you start to change it."
Though journalists struggle to use data to quantify the impact of their own work, other members of the panel argued that the increasing popularity of data journalism and digital literacy has the potential for a broader impact on society as a whole.
Author and journalist Dan Gardner was critical of some of the backlash to the popularity of data journalism. "The problem that you face in the crafting of public policy is not that it is too focused on empirical evidence, it is quite the opposite, it is let us ignore the empirical evidence, except when it happens to coincide with our biases." Overemphasizing anecdotal examples in news articles could result in story that is unrepresentative of the facts at hand, he warned. But he also emphasized that it was important to complement empirical reports with insights gained from talking to people that can "surface the really subterranean questions" motivating peoples' opinion, such as their moral beliefs about marijuana legalization, even when supposedly fact-based arguments are dominating the public discussion.
He emphasized that the popularity of data-driven analysis had sped up the ability of science and journalism to not only arrive at a consensus, but also to act as a check on authority.
"It is not one data journalist who will be authoritative and find the truth, it is that great collective argument that eventually thrashes out the truth," he said. That process has led journalism and the public to be more effective and successful at "calling bullshit on politicians" through fact checking websites and other platforms. "I think there are fewer stories about the effects of the communication strategy, what this means for the Clinton campaign ... and when a politician really tells a whopper, they really do suffer for it and they get chased for it and they haven't in the past."
For Mark Hansen, director of the Brown Institute for Media Innovation at the Journalism School for new data journalism skills "to call bullshit on politicians" has larger implications at the K-12 education level beyond journalism school. He suggested that it was necessary to change the teaching of statistics from being "more like a math class " in order to "hybridize something like journalistic education ... along with data training at the K-12 level, so you have sufficient scaffolding that the students can learn about climate change, can learn about obesity, can learn about bullying in schools, can talk about all these things, that they can't talk about in current probability and statistics classes because they are taught by algebra teachers...there needs to be place where you can talk about socially relevant issues in the context of data...."
As part of the conference, Tow Center researcher Alex Howard also presented a report on the best practices in data journalism, Tow Research Scholar Fergus Pitt and several contributors explored the evolution of sensor journalism, while researchers Claire Wardle and Sam Dubberley examined the use of user generated content by TV and online news outlets.
Fellows Brian Abelson and Michael Keller presented their NewsLynx project, which aims to create a platform to help news outlets explore the qualitative impact of investigative news by drawing on a wide range of analytics, and researcher Andy Carvin previewed his research into how false reports spread on social media and mainstream news outlets during reporting on the Gabrielle Giffords shooting, the Newtown Creek shooting and during the aftermath of the Boston Marathon bombing and ways journalists could influence those social media reactions as they unfold.