Personal Democracy Plus Our premium content network. LEARN MORE You are not logged in. LOG IN NOW >

[BackChannel] Why You Should Test (Almost) Everything

BY Benjamin Simon and Jim Pugh | Monday, May 6 2013

techPresident's Backchannel series is an ongoing conversation between practitioners and close observers at the intersection of technology and politics. Jim Pugh is the CEO of ShareProgress. He previously ran the digital analytics program at Organizing for America.
Ben Simon was formerly the director of new media campaigns at the DNC and OFA, and he is currently working as an independent consultant.

One of the best things to come out of the post-campaign coverage of OFA 2012 has been a renewed focus on analytics -- and in particular on randomized testing and experimentation -- as a crucial part of any good digital program. It’s something we’ve both been preaching for years, and we're excited to see its growing proliferating outside of just the largest programs.

Randomized testing is an incredibly valuable tool. It lets you use data to determine which messages resonate the most and drive your supporters to take action, rather than needing to make guesses based on your gut instinct (which we can attest will often be inaccurate). Applying the results of these tests can increase the impact of your digital program by a substantial margin.

However, it's important to recognize that there's an opportunity cost associated with any test you run. Even the simplest email subject line test requires time and effort to plan and execute. And more complicated tests take even more work -- to execute a 4x3 email test (four different emails with three subject lines each) requires writing four separate drafts, coding each one up separately, and analyzing a lot more response data.

There may be a credibility cost to testing as well. Many organization directors are still skeptical about the value of testing, so every experiment that takes time without yielding useful results could be a strike against the cause of testing more generally.

A good test can be well worth it -- and pay off handsomely by increasing the impact your program. But to gain useful, actionable results, your experiment needs to provide you with enough data to see statistically significant differences between the approaches that you're testing -- ideally with 95% confidence or more.

If you're the Obama campaign, MoveOn.org, or Avaaz and have an email list of millions of people to contact, it won't be hard to collect enough responses to reach this threshold. But for smaller organizations with more limited reach, it may be much more difficult.

How can you check in advance to see if you'll reach statistical significance? Here's what you need to do:

  1. Identify what it is you're trying to maximize. If this is an email, it should be your ultimate action (donations or petition signatures, for example), rather than simply opens or clicks. If it's a webpage test, it should be whatever action you want people to take on the page.
  2. Figure out how many people you intend to reach with each different approach that you're testing.
  3. Estimate, based on past performance, what percentage of the people you're reaching you expect to take action. This percentage will be very different depending on what your action is (for example, you'll probably have a much lower action percentage when asking people to make a donation than when asking them to sign a petition).
  4. Make an educated guess about how much difference in response you might see between your approaches. Will one version get 5% more actions than another? 10%? 30%?
  5. Using your estimates from the previous steps, calculate how many actions you expect for each of your different approaches, then plug your reach and action numbers into a statistical significance calculator to measure the expected confidence level of your results.

Is the difference between your approaches significant? If yes -- go ahead with your test! There's a good chance your results will be able to tell you which approach is best.

But if the answer's no? Then don’t run the test. Without statistical significance, it's very possible that the approach with the highest number of actions may not actually be the best one, and the test therefore doesn't provide you with useful information. The time it took to set up and run the experiment could have been more effectively spent elsewhere.

News Briefs

RSS Feed tuesday >

Ruck.us Reboots As a Candidate Digital Toolkit That's a Bit Too Like Democracy.com

Ruck.us launched with big ambitions and star appeal, hoping to crack the code on how to get millions of people to pool their political passions through their platform. When that ambition stalled, its founder Nathan Daschle--son of the former Senator--decided to pivot to offering political candidates an easy-to-use free web platform for organizing and fundraising. Now the new Ruck.us is out from stealth mode, entering a field already being served by competitors like NationBuilder, Salsa Labs and Democracy.com. And strangely enough, Ruck.us seems to want its early users to ask Democracy.com for help. GO

Armenian Legislators: You Can Be As Anonymous on the 'Net As You Like—Until You Can't

A proposed bill in Armenia would make it illegal for media outlets to include defamatory remarks by anonymous or fake sources, and require sites to remove libelous comments within 12 hours unless they identify the author.

GO

monday >

The Good Wife Looks for the Next Snowden and Outwits the NSA

Even as the real Edward Snowden faces questions over his motives in Russia, another side of his legacy played out for the over nine million viewers of last night's The Good Wife, which concluded its season long storyline exploring NSA surveillance. In the episode titled All Tapped Out, one young NSA worker's legal concerns lead him to becoming a whistle-blower, setting off a chain of events that allows the main character, lawyer Alicia Florrick (Julianna Margulies), and her husband, Illinois Governor Peter Florrick (Chris Noth), to turn the tables on the NSA using its own methods. GO

The Expanding Reach of China's Crowdsourced Environmental Monitoring Site, Danger Maps

Last week billionaire businessman Jack Ma, founder of the e-commerce company Alibaba, appealed to his “500 million-strong army” of consumers to help monitor water quality in China. Inexpensive testing kits sold through his company can be used to measure pH, phosphates, ammonia, and heavy metal levels, and then the data can be uploaded via smartphone to the environmental monitoring site Danger Maps. Although the initiative will push the Chinese authorities' tolerance for civic engagement and activism, Ethan Zuckerman has high hopes for “monitorial citizenship” in China.

GO

The 13 Worst Bits of Russia's Current and Maybe Future Internet Legislation

It appears that Russia is on the brink of passing still more repressive Internet regulations. A new telecommunications bill that would require popular blogs—those with 3,000 or more visits a day—to join a government registry and conform to government-mandated standards is expected to pass this week. What follows is a list of the worst bits of both proposed and existing Russian Internet law. Let us know in the comments or on Twitter if we missed anything.

GO

Transparency and Public Shaming: Pakistan Tackles Tax Evasion

In Pakistan, where only one in 200 citizens files their income tax return, authorities published a directory of taxpayers' details for the first time. Officials explained the decision as an attempt to shame defaulters into paying up.

GO

wednesday >

Facebook Seeks Approval as Financial Service in Ireland. Is the Developing World Next?

On April 13 the Financial Times reported that Facebook is only weeks away from being approved as a financial service in Ireland. Is this foray into e-money motivated by Facebook's desire to conquer the developing world before other corporate Internet giants do? Maybe.

GO

The Rise and Fall of Iran's “Blogestan”

The robust community of Iranian bloggers—sometimes nicknamed “Blogestan”—has shrunk since its heyday between 2002 – 2010. “Whither Blogestan,” a recent report from the University of Pennsylvania's Iran Media Program sought to find out how and why. The researchers performed a web crawling analysis of Blogestan, survey 165 Persian blog users, and conducted 20 interviews with influential bloggers in the Persian community. They found multiple causes of the decline in blogging, including increased social media use and interference from authorities.

GO

More