A Night of Open Government at the New York Times

Last night on the 15th floor of the New York Times Building in Manhattan, the Times' in-house team of data-minded folk held another public event in their "Times Open 2.0" series, this one on the topic of open government. Folks presented, questions were asked, and the long and the short of it: we've just passed the "Hello, World" moment in the evolution of open government. The tools have been, if not mastered, at least learned. In lieu of answers, there are some hard-earned questions about where things go from here. Some notes, in that vein, of last night's presenters, what they've learned, and where they're headed next:

• Big Apple Ed

  Adda Birnir, J. Soma, and Kate McGee Reyes built Big Apple Ed, the third place winner in New York City's Big Apps contest, launched to draw attention to the then-new NYC Data Mine. "The nut of the site," said Birnir, "is that it provides a profile for each of the 1,600 schools in New York City." The team picked education out of a desire to work on something meaty. "It's controversial," said Birnir. "It's important." Soma suggested that developers have some humility about the worth of the data they have at their finger tips, suggesting that a could baseline of thought for apps developers is that, "your data is dirty and your algorithms are wrong." Other lessons learned: Birnir said that having a diverse team can help in figuring out what the data is telling you, and Soma said he's found that the "data hoarders" working inside government tend to be normal folk who just haven't fully gotten on the open-data train yet. Next up for the team is, they hope, working with journalists to ferret out stories in the data. Soma wrapped by putting in perspective the modern age's relative flood of government data, and what it means for developers. "It's an amazing responsibility," he said, "and an awesome power."

• One Bus Away

  Nick Grossman, Director of Civic Works at OpenPlans (formerly The Open Planning Project) talked about his organization's work in attempting to help New York City figure out how to build an affordable, do-able real-time bus API. One way to do it: hire the guy or gal who already figured out how to do it elsewhere. That's what they've done, by bringing on one of the creators of Seattle's One Bus Away app and charging him with adapting it to Gotham. The ultimate goal, said Grossman, was to build an Open Platform for Transportation, such that cities won't have to re-invent the wheel every time they want to track their buses or monitor their parking meters. Want to help? The transit hackers group is at Groups.Google.com/group/transit-developers. You might also dive into the conversation through #opentransit on IRC.freenode.net.

• Also of the rinse-and-repeat mindset is James Turk, a developer and open source coordinator for Sunlight Labs*. Turk discussed the Open States Project (and his slides are here.) The problem, said Turk, is that there's no GovTrack.us for the states of the United States, no hub that integrates, as GovTrack does on the federal level, the legislative feeds coming out of the various legislative bodies and resources. There are more than a hundred state legislative websites, said Turk, and Sunlight is aiming to rally volunteer developers to make some sense of them. One thing going their way is that often one state, once tackled, can serve as a model for another state -- making Open States a good gateway project for aspiring open-source developers. The work flow is basically thus: figure out how to scrape data from state websites without pulling them down, blending together different datasets to present the fullest possible legislative picture, and then pushing it out to the world through APIs or data dumps.

• Frank Macreery and Matthew Beale built PushpinWeb, which got an honorable mention during the Big Apps contest. Beale placed himself and Macreery in the "weird gooey middle" of development shops as a two-person team. And the landscape for folks like them, he suggested, was occasionally rough. The biggest monster: "hostile datasets" being pushed out by government. What makes a dataset so adversarial to the go-it-alone developer? Undocumented file formats ("binary even," bemoaned Beale), proprietary file formats like ArcGIS, missing update schedules, and the sheer bulk of some government datasets -- the last of which, said Beale, "really spooks off casual developers." One way to make the job simpler, they found is to stick with the tools that you know, which for them meant relying upon Ruby on Rails, MySQL, and Sphinx to build their app.

• The Trees Near You app

  Finally, Brent Camper, who built Trees Near You, another Big Apps honorable mention honoree. Camper admitted no particular affinity for trees, but saw in the relevant city data sets a way to use visualizations to present a different, greener view that even New Yorkers might have of their fine city. Using your location, Trees Near You tells you, yes, the trees near you, with insights into which species they are and how big they were at last measure. Camper struggled with inconsistencies between datasets of different boroughs, and with the lack of documentation that came along with the released data. Both were overcome with a lot of work and a little luck; what the species codes in the data meant was a mystery until he found a key in the Google cache of the New York City Parks Department website. For other folks looking to build map apps off of civic data, Camper recommended both a tool set -- OpenStreetMap, plus MapNik to render map tiles and Cascadenik for styling those maps, with TileDrawer to serve the whole thing up -- and reading material: Seth Fitzsimmons' "OS X Spatial Stack."

Next up in the TimesOpen 2.0 series is Wednesday, October 6th, on the "real-time web."

*Note: Our Andrew Rasiej and Micah Sifry are senior advisors to the Sunlight Foundation, which runs Sunlight Labs.