Personal Democracy Plus Our premium content network. LEARN MORE You are not logged in. LOG IN NOW >

The Trouble with Salsa: Inside Yesterday's Dem Software Platform Woes

BY Nancy Scola | Thursday, May 20 2010

Credit: CaptPiper

Salsa Labs is, today, reconsidering having one line of connection through the Internet between its software-as-a-service platform and the many progressive and Democratic groups that uses its messaging and organizing software. At least one client, though, is wondering if that examination is overdue. "We left a lot of money on the table yesterday," said a representative from one Democratic organization, a Salsa client.

The trouble is that yesterday, the day after some fairly major primary races in the U.S., Salsa Lab's eponymous flagship software project experienced a complete outage lasting at least five hours that left its many clients totally unable to use the platform. Salsa's client list, a sort of who's who of progressive politics, includes the AFL-CIO, ActBlue, Chuck Schumer's re-election campaign, the DCCC, and many others.

Justin Nemmers, Salsa Labs' Chief Operating Officer took responsibility for the problem, saying, "We should have done better, we could have done better, we will have done better."

The nuances of exactly what went wrong with Salsa yesterday are, frankly, complicated, and a little cloudy at this point. Your faithful blogger spent a few hours today going down the rabbit hole of multiply redundant Internet service providers, fiber optic back haul, Border Gateway Protocol and other, well, excruciating topics.

But here's what we know

Salsa's Nemmers, while, again, acknowledging that Salsa Labs let its customers down yesterday, traced the problem to the company's London-by-way-of-LA Internet service provider, Packet Exchange.

"The situation," said Nemmers, who previously managed government services at the open-source software company Red Hat, "was that in working with our ISP, we were under the impression, we had been told that we were multiply redundant," or, in other words, that Salsa Labs were paying their ISP for more than one path to the rest of the Internet. Should one footpath from Salsa to the rest of the world get washed out, the thinking goes, the other paths should remain open and viable. "We've never had a reason to believe otherwise," said Nemmers, about his expectation of redudancy. "We've never had an interruption like this. We've taken all the best practices and reasonably accepted steps. But a bunch of things came together and ended up causing us to go down."

That 'bunch of things,' says a PR rep for Packet Exchange, Salsa's ISP, traces back to a FiberLight, the Georgia-based fiber optic cable company that Packet Exchange relies upon for its Internet service in the Washington DC area. According to the rep, a construction accident yesterday at a Reston, Virginia data center ended up with a cut FiberLight cable, which cascaded down until Salsa Labs found itself without service. "Outages happen," said the rep. "Fibers get cut." Digging, for example, can often rip up existing cables. One of the worries floating around the cyber security world is that the deliberate targeting of the cables that together form the Internet could leave Americans disconnected as Salsa customers were yesterday. A rep for FiberLight did not return a request for comment left earlier today.

Packet Exchange's rep was not able to respond immediately to questions about Salsa's expectations of multiply redundant service from the company. Technicians at both Fiber Light and Packet Exchange, she said, are still digging through yesterday's events to figure out what went wrong.

Nemmers, for his part, says that Salsa Labs isn't taking chances from here on out. By the end of the day today, or tomorrow morning at the latest, said the COO, Salsa Labs will have up and running two completely discrete connections to the Internet. Internet routing work, including the use of the Border Gateway Protocol mentioned above, will blend the two systems together. "We'll be in a situation where if there was a second interruption there would be no interruption to our customers," said Nemmers. "We're working until it's done, and we're not stopping until it's done."

Because Salsa's own email system was connected to the problematic ISP, Salsa Labs wasn't able to reach its customers by email yesterday to discuss the troubles. The company found another path. Said April Pedersen, Salsa Labs co-founder, reached via email today, "We directed all of our pages to our Twitter stream and provided real-time updates for the duration of the outage."

Related posts: "Spicy Industry News: AFL-CIO Taps Salsa"

News Briefs

RSS Feed wednesday >

Facebook Seeks Approval as Financial Service in Ireland. Is the Developing World Next?

On April 13 the Financial Times reported that Facebook is only weeks away from being approved as a financial service in Ireland. Is this foray into e-money motivated by Facebook's desire to conquer the developing world before other corporate Internet giants do? Maybe.


The Rise and Fall of Iran's “Blogestan”

The robust community of Iranian bloggers—sometimes nicknamed “Blogestan”—has shrunk since its heyday between 2002 – 2010. “Whither Blogestan,” a recent report from the University of Pennsylvania's Iran Media Program sought to find out how and why. The researchers performed a web crawling analysis of Blogestan, survey 165 Persian blog users, and conducted 20 interviews with influential bloggers in the Persian community. They found multiple causes of the decline in blogging, including increased social media use and interference from authorities.


tuesday >

Weekly Readings: What the Govt Wants to Know

A roundup of interesting reads and stories from around the web. GO

Russia to Treat Bloggers Like Mass Media Because "the F*cking Journalists Won't Stop Writing"

The worldwide debate over who is and who isn't a journalist has raged since digital media made it much easier for citizen journalists and other “amateurs” to compete with the big guys. In the United States, journalists are entitled to certain protections under the law, such as the right to confidential sources. As such, many argue that blogging should qualify as journalism because independent writers deserve the same legal protections as corporate employees. In Russia, however, earning a place equal to mass media means additional regulations and obligations, which some say will lead to the repression of free speech.


Politics for People: Demanding Transparent and Ethical Lobbying in the EU

Today the Alliance for Lobbying Transparency and Ethics Regulation (ALTER-EU) launched a campaign called Politics for People that asks candidates for the European Parliament to pledge to stand up to secretive industry lobbyists and to advocate for transparency. The Politics for People website connects voters with information about their MEP candidates and encourages them to reach out on Facebook, Twitter or by email to ask them to sign the pledge.


monday >

Security Agencies Given Full Access to Telecom Data Even Though "All Lebanese Can Not Be Suspects"

In late March, Lebanese government ministers granted security agencies unrestricted access to telecommunications data in spite of some ministers objections that it violates privacy rights. Global Voices reports that the policy violates Lebanon's existing surveillance and privacy law, Law 140, but has gotten little coverage from the country's mainstream media.


friday >

In Google Hangout, NYC Mayor de Blasio Talks Tech and Outer Borough Potential

New York City Mayor Bill de Blasio followed the lead of President Obama and New York City Council member Ben Kallos Friday by participating in a Google Hangout to help mark his first 100 days in office, in which the conversation focused on expanding access to technology opportunities through education and ensuring that the needs of the so-called "outer boroughs" aren't overlooked. GO

thursday >

In Pakistan, A Hypocritical Gov't Ignores Calls To End YouTube Ban

YouTube has been blocked in Pakistan by executive order since September 2012, after the “blasphemous” video Innocence of Muslims started riots in the Middle East. Since then, civil society organizations and Internet rights advocacy groups like Bolo Bhi and Bytes for All have been working to lift the ban. Last August the return of YouTube seemed imminent—the then-new IT Minister Anusha Rehman spoke optimistically and her party, which had won the majority a few months before, was said to be “seriously contemplating” ending the ban. And yet since then, Rehman and her party, the conservative Pakistan Muslim League (PML-N), have done everything in their power to maintain the status quo.


The #NotABugSplat Campaign Aims to Give Drone Operators Pause Before They Strike

In the #NotABugSplat campaign that launched this week, a group of American, French and Pakistani artists sought to raise awareness of the effects of drone strikes by placing a field-sized image of a young girl, orphaned when a drone strike killed her family, in a heavily targeted region of Pakistan’s Khyber-Pakhtunkhwa Province. Its giant size is visible to those who operate drone strikes as well as in satellite imagery. GO

Boston and Cambridge Move Towards More Open Data

The Boston City Council is now considering an ordinance which would require Boston city agencies and departments to make government data available online using open standards. Boston City Councilor At Large Michelle Wu, who introduced the legislation Wednesday, officially announced her proposal Monday, the same day Boston Mayor Martin Walsh issued an executive order establishing an open data policy under which all city departments are directed to publish appropriate data sets under established accessibility, API and format standards. GO

YouTube Still Blocked In Turkey, Even After Courts Rule It Violates Human Rights, Infringes on Free Speech

Reuters reports that even after a Turkish court ruled to lift the ban on YouTube, Turkey's telecommunications companies continue to block the video sharing site.