NY Study Shows How Freedom of Information Can Inform Open Data
BY Miranda Neubauer | Thursday, August 14 2014
On New York State's open data portal, the New York Department of Environmental Conservation has around 40 data resources of varying sizes, such as maps of lakes and ponds and rivers, bird conservation areas and hiking trails.
But those datasets do not include several data resources that are most sought after by many New York businesses, a new study from advocacy group Reinvent Albany has found. Welcome to a little-discussed corner of so-called "open government"--while agencies often pay lip service to the cause, the data they actually release is sometimes nowhere close to what is most wanted.
Reinvent Albany used the Freedom of Information Law (FOIL) process to obtain the logs of the nearly 4,000 FOIL requests to the agency in 2013, and found that half a dozen datasets were the subject of fifty percent of all requests, including "two specific datasets about oil spills [that] aren't online anywhere," explained Dominic Mauro, staff attorney for the group, in a recent interview.
Since the FOIL logs contain information about who is requesting them, Reinvent Albany was able to take a closer look at the interests of those requesting information.
According to their analysis, over 80 percent of all requests were from business interests, with journalists and advocates together making up another 5 percent of the requests. Some of the most frequent "FOILers" were environmental consultants who advise lawyers and realtors who are responsible for researching the impact of oil spills on properties, Mauro explained.
Reinvent Albany reached out to two real estate firms that had filed a total of 300 FOIl requests in 2013. "The firms both described the current DEC open data offerings as not meeting the needs of the real estate industry," the report states. "For instance, spill incidents can only be found on the DEC’s website via a search form that allows one location to be searched, one year at a time. DEC spill incident reports date back to 1978, so users would need to run 37 separate searches to find all spills for a specific address. Users who have an incident number can get look up rudimentary information about individual spills."
The existing database and search function only provides the name of the chemical that was spilled for a location, not when it was cleaned, inspected or received a health certification, Mauro pointed out.
"What we're seeing over and over again is agency officials guessing what is high value...We're just not seeing coherence between the FOIL logs and what DEC [puts online]," he said. "How are they determining what high value means?" The question echoes a similar debate at the New York City level, where advocates have also increasingly emphasized quality and value over quantity.
In the view of Reinvent Albany, the DEC could reduce FOIL requests by 50 percent by publishing frequently FOILED data.
While DEC's online offerings fall short, the agency ironically does a very good job of cataloging Freedom of Information requests. In its report, Reinvent Albany notes that its "analysis was only possible because DEC keeps careful, digitized, FOIL logs" unlike many other agencies.
Mauro said it shouldn't be difficult for an agency like the DEC to analyze FOIL requests in a more systematic way in connection with open data.
"We don't have a lot of resources here, I'm just one guy ... I don't have a background in environmental science, if I can do this without the expertise that somebody has, like a FOIL officer, who knows about the subject matter, I should think it's possible [for the agency]," he said.
Noting that the DEC handles more FOIL requests than any other agency, Mauro suggested that it had the potential to become the next "marquee open data agency" in the state by following the lead of the New York State Department of Health, which has won awards for its open data policy with one of the "best FOIL regimes" and an approach that is "all metrics based." He said it would it make sense to focus on the open data policy of different state agencies every couple of months over the year, in the spirit of Governor Andrew Cuomo's 2013 Executive Order 95, which called on state agencies to make their data available through the Open NY platform.
"Open Data is a routine part of DEC’s work and is a focus at the agency’s executive level who work closely with Open Data NY," DEC spokesperson Dianne Patterson said in an e-mailed statement, noting the over 40 data resources that the agency made has made available on Open NY. "DEC follows the guidance outlined in the New York State Open Data Handbook in determining what data to post on the Open Data website, as well as in prioritizing data sets for ongoing and continuous publication." She also added that "agency staff are vigilant about communicating with peers in other agencies to identify additional ways to be successful, and continually examine our extensive data set review process to ensure publication does not expose individuals or businesses to harm."
In March, Cuomo released the one-year report for New York's Open Data portal, which notes that nearly 800 catalog items were available on the portal in the second quarter of 2014, up from 244 in March 2013. But a graphic in the report also shows that 90 percent of the DEC catalog items have to do with recreation, while only about ten percent have to do with energy and the environment. The March 2014 press release notes that the state plans to release a new transparency hub in the near future, incorporating ideas from a workshop with open government advocates.
In a webinar last year, then New York Health Commissioner Nirav Shah discussed how that agency made a concerted effort to be more proactive about releasing data. "We targeted the datasets where we were getting lots of Freedom of Information Law requests," often numbering between 5,000 and 7,000. "In the first three months alone, we eliminated 700 of those requests." He also spoke about the culture change necessary for shifting towards more openness. "One of the concerns is, well, if we release this data, it will make us look bad. And yeah, that's the point. There are opportunities where you may need to fix things because the data shows trends or deficiencies in the work of your own state agencies," he said. " That is a conscious choice you have to make. We made that choice, it took some time for us to get there because we showed that the ultimate value was much greater than holding that data back."