You are not logged in. LOG IN NOW >

Ushahidi Provides Journalists With Instant Real-Time Crisis Data

BY Onnik James Krikorian | Wednesday, August 20 2014

Ushahidi's CrisisNET platform provides reporters with accurate and timely data culled from social media (credit: Ushahidi)

“Wouldn’t it be great if we had an algorithm that was smart enough to read the world news and anticipate conflict before it happens,” says Jonathan Morgan to techPresident. He is the Co-Founder and Technical Director for Crisis.Net, a new Application Program Interface (API) developed by Ushahidi. “But that’s not just a limitation of technology,” he says.

“There are just too many variables and it’s important to have deep domain expertise of the area, the groups and ethnicities engaged in conflict, and even the topology of the region,” Morgan explains. “There’s a massive amount of knowledge that’s required to truly understand how a conflict is unfolding that's quite specific to a particular region.”

But times have changed since Ushahidi first launched its crisis mapping platform in the violent aftermath of the 2007 elections in Kenya. With the use of social media now widespread, so too has the way in which many media, international organizations, and local NGOs now work. Ushahidi has had to go social.

Consuming already existing data can prove quite a hurdle for many journalists and analysts even without having additional sources added into the mix. Even if experienced in doing so, they often have to relearn and adapt the same process when they move to a crisis or conflict in another region. And those not used to retrieving and using data have an even steeper learning curve.

“That’s the real barrier to acting quickly when conflicts or other emergency situations arise,” Morgan concludes. “Some data will already exist while other data is generated in real-time from streaming information sources such as social media, sensors, or even quasi-real-time data that comes out of traditional media, RSS feeds, and stuff like that.”

And that’s where Crisis.Net comes in with its ability to deliver data pulled from a variety of sources in Javascript Object Notation (JSON) or Comma-Separated Values (CSV) formats. The API should make it possible for media organizations and developers alike to build their own applications without the need to spend days locating, identifying, and processing data.

“What we found with Ushahidi was that we had to rely on information directly submitted by users interested in a particular crisis, disaster, or a conflict, but that when we wanted to do any analysis on data on a particular conflict, in the humanitarian space at least, it was tucked away in tiny corners scattered all over the Internet,” says Morgan.

“There might be a non-profit doing some work in a particular region or there’s collected data stashed in reports sitting on a server somewhere,” he explains. “Or maybe there’s a list of reports in a spreadsheet on Google Drive or a hashtag that we want to follow for information relevant to a crisis, but there’s no good way to collect all of this disparate data in a single place.”

To demonstrate the potential of Crisis.Net, the Facebook pages of Palestinian supporters were sourced to visualize recent attacks on Gaza. Tweets have also been analyzed to assess attitudes to the Israeli Defense Force (IDF). Crisis.Net has also created a timeline charting the Islamic State (of Iraq and the Levant) in Northern Iraq based on YouTube, Twitter, and Facebook conversations.

A screenshot of a Crisis.Net visualization showing public sentiment towards IDF on social media (credit: Ushahidi)

Partnering with AlchemyAPI, facial recognition technology was used to filter content of IS members from Twitter. But, as its name suggests, Crisis.Net is not just about conflict.

“The World Policy Journal is also building on a map based on data we retrieved for them about the spread of Ebola in West Africa,” says Morgan, “and we’ve also been in discussion with the African Union and some other international NGOs about combining different data for early warning systems that’s more humanitarian focused.”

A coalition of civil society, government and international agencies used the Ushahidi platform to create an online Ebola monitoring system in Liberia (credit: Ushahidi)


As with Ushahidi itself, verification is vital, but that is left up to end users. It’s also why Morgan worked with Eliot Higgins, better known online as Brown Moses. Working on the conflict in Syria by monitoring social media sites, Higgins had already managed to put together, collect, and collate precise information about militant groups in the conflict.

Working with the 1,700 Facebook pages and YouTube channels Higgins monitors and piping them into Crisis.Net, Morgan was able to demonstrate that maps of the conflict derived from information on social media could be as accurate if not as detailed as those produced by the BBC using data manually collected by humanitarian organizations working in the region.

Data collected via social media matched what was produced by the BBC through ground reports.

“We’re not trying to be the arbiters of truth or what conflict or crisis data is,” he explains to techPresident. “There might be interest in highly subjective content from a particular community in order to examine exactly that kind of polarization. We also try to attach enough metadata so that users can determine whether or not a piece of information is valid for the sort of analysis they want.”

“What’s the right level of abstraction?” he muses. “At Ushahidi in general we work hard to make platforms that make just enough choices to make sure that we’re building tools that are useful for as broad an audience as possible, but without making so many choices that we’re limiting the possibilities of the platform as well.”

Language is another problem.

“Machine translation is not useful for understanding what somebody is trying to say,” he says, “but enough of marker keywords are consistently translated. Things like ‘killing’ or ‘barrel bombs,’ an important one in Syria, are pretty reliably translated and enough to give our algorithms what they need in order to categorize content and understand the names of places or people.”

It also depends on the situation in terms of social media use itself. Nigeria, for example, is a ‘black hole,’ according to Morgan while there’s an abundance of sources in the Middle East.

One issue still being determined, however, is the interface and whether it should be made drag and drop so that users don’t need any technical expertise to interact with raw data. Crisis.Net is also slated to be incorporated into the third major release of Ushahidi in order to allow its own visualizations to be populated by data derived from the API.

Morgan is also advocating for international organizations to become more open in their data policies. For now, much of it is locked away and only some data is released publicly. This is a valuable resource in its own right, he explains, but especially when reliable information from social media is absent or completely lacking.

“I’m always surprised by the stories a dataset, or a different combination of datasets, can tell and I don’t think there’s an appreciation of that in those large organizations about the potential power for change,” notes Morgan. “They really have a valuable resource, but they’re just not very receptive to the idea that it can be shared publicly in a way that’s safe.”

Ushahidi hopes that Crisis.Net will encourage others to build their own applications with its API. “The long-term goal of the platform is to make data so accessible that those with beginner to intermediate technical expertise can build their own applications to benefit their communities instead of, say, another app to share photos with friends,” he says.

“I think that people are really attracted to that idea, but have no place to start. There’s just no way for them to do so without having a lot of knowledge about where all that information can come from. I hope we can move people to a higher level of social engagement if we can make the repetitive process of finding and combining information sources easier.”

Onnik James Krikorian is a journalist, photographer, and media consultant from the United Kingdom. From 2007-2012 he was the Caucasus Regional Editor for Global Voices.

Personal Democracy Media is grateful to the Omidyar Network for its generous support of techPresident's WeGov section.