Under Open Data Law, New York City Begins Herding Its Data
BY Miranda Neubauer | Monday, March 11 2013
New York City had until last Thursday to meet the first deadline set in its now year-old open data law by making data already published on nyc.gov available in machine-readable format, rather than in PDF format.
According to a city press release, there are now over 1,000 data sets available on New York City's Open Data platform. The platform launched in October of 2011 with 750 data sets, 250 of which were new at the time. Since the law was signed in March of last year, New York City's Department of Information Technology and Telecommunications (DoITT) has been working with agencies to add 350 new data sets to the platform and worked to add regularly updated feeds to existing data sets.
Prior to last week's deadline, open government advocates had said that they did not expect all agencies to meet the deadline, but that they were encouraged by the progress made so far, especially given some delays that were due to Superstorm Sandy. Advocates also said that more legislation would be necessary to expand on and further implement the law.
Under the next milestone in the beginning of September, agencies must present a plan to release on the Internet all data that is already available for public review, provide an explanation about any data that cannot be made available, and set a timeline for the release of additional data by 2018 at the latest.
Andrew Nicklin, director of research and development at DoITT, said that one challenge in working with agencies is simply navigating the government structure, locating the data and contacting the people who manage that data. But he noted that one aspect that hasn't been challenging is "getting people [within the city] excited about it." Nicklin attributed that enthusiasm to the fact that "the Mayor has made this a priority [and that] people are starting to see value in it."
In some cases making data available through the platform has had other benefits for city operations. A Buildings Department website was getting so much traffic due to data requests that "we were worried about internal staff not being able to do their job," he said.
"To mitigate that, we took some of the data and began delivering it through the Open Data Portal," he explained, "which reduced the amount of of traffic on our main site and off-loaded it to the open data platform."
Another challenging aspect of the process for the city is determining what datasets are really in demand by the public, he said.
"We are struggling to recognize the data that is valuable versus what is not valuable," he said. "There is some data out there that maybe nobody is interested and we have spent time and energy putting it out there."
As the process continues, he said he hoped the city can get better at figuring out which data sets might be worth leaving for later and which more important ones to devote more resources to.
One existing way of helping with that determination is the slightly hidden Suggest a Dataset feature, accessible at the bottom of the platform's front page and through the FAQ, where users can submit datasets they are interested in and vote up existing suggestions.
So far, only a few requests have between two and four votes including a request for electrical permits from the Buildings Department, data on land records and property transactions from the City Register and Department of Buildings data related to permits, inspections and violations.
Jack Wright, a user experience designer for a digital agency, posted that he was working on iPad app with some friends to show information for New York City neighborhoods, and was interested in an API or a data set showing crimes committed in the city.
One user requested data showing GIS coverage of sewer outfalls into city surface water. Nicklin replied that the city would see what it could do, but "it's likely we won't be able to release this data - my understanding is that it is considered 'sensitive.'"
Other queries requested data on the locations of NYC fire hydrants for a New York City version of Code for America's Adopt a Hydrant program and the percent of 311 calls requiring foreign-language interpretation.
Among the recently released datasets that the city highlighted is historical city-wide New York City crime data for the past 12 years for the seven major felonies, as well as non-major felony crimes, misdemeanors and violations and a new 10-year view of more than 1,200 Mayor’s Management Report indicators.
Other data added last week includes an overview of stalled construction sites, issued permits and complaints from the Department of Buildings that will be updated daily beginning April 10. But in all three instances, the entry notes that previous daily and weekly reports are archived at DOB and not available through NYC Open Data.
Nicklin said he interpreted the law in a very broad way, with its mandate applicable not only to entities such as the NYPD,the Department of Education, and other City Hall agencies, but also the District Attorney's office, the courts, the City Council, the Public Advocate, the City Comptroller, the borough presidents, and the local community boards. The law would not apply to the MTA, which is a state agency and has its own open data initiatives.
In the next few months, Nicklin said he expected major data sets with "high value" to be added as part of the initiatives overseen by the city's new Director of Analytics, Mike Flowers.
Meanwhile, DoITT is turning its focus on the compliance plans due in September. He said he expected that agencies would receive a template to fill out asking them to inventory their internal systems and centralized databases and identify which can and should be made public. DoITT would then work the agencies as the lists are refined and undergo a series of reviews.