You are not logged in. LOG IN NOW >

If Data Is The First Step, Scraperwiki is a Staircase

BY Nick Judd | Wednesday, December 8 2010

One of the tools that made a splash during the International Open Data Day hackathon last weekend was ScraperWiki, a platform, new this year, for building and running scrapers to free public data from clunky or poorly designed government database interfaces.

"Just the specificity offered by Scraperwiki is really cool," Max Ogden, an incoming Code for America fellow, PDXAPI author and owner of a singular beard, told me by phone earlier this week. "The website's about the data and not about the code."

Ogden co-organized the Portland, Ore. edition of that international hackathon, which turned into a bit of a showcase for Scraperwiki. Using Scraperwiki allowed Ogden and Selena Marie Deckelmann, a consultant and database developer, to quickly glean data from public databases and use it to build visualizations that help track business creation in Portland, and lay a framework to track mergers and acquisitions.

"The mayor's office is trying to develop the software industry in Portland," Ogden told me.

But at a recent summit on the software industry, Deckelmann writes, she was told that it's hard to keep track of movements in that sector of Portland's economy. So she used Scraperwiki to try and get a better idea of what was going on.

She and Ogden were just two of many people at the hackathon working on scrapers, though. By providing a platform that would automatically continue to run the scrapers that programmers build on the site, and store the resultant data at no cost, Scraperwiki eliminated the need for newbies to spend time setting up their computers to run scrapers in the first place, Ogden said.

ScraperWiki is a project of some of people at MySociety, sort of the older uncle of the U.S.' Sunlight Foundation, created to solve exactly that problem: In order to build the kinds of applications that MySociety is well known for building — the legislator-accountability tool TheyWorkForYou, for example — the information those tools help you find first needs to be machine-readable.

"The data is a necessary first step," Francis Irving, a former MySociety developer and the CEO of Scraperwiki, told me yesterday.

The service launched in March and has been going through constant development since, Irving said. It's free to have Scraperwiki scrape data that anyone can then see, but the company — organized as a for-profit — also offers to run private scrapers, such as for doing data migration inside a firm's internal intranet.