Testing New Search Tools on Government & Campaign Information
BY Sarah Granger | Wednesday, December 3 2008
Back in the day, when Yahoo! was the only search game in town, many wondered why Ask Jeeves (now Ask.com), and eventually Google would attempt to break into that market. The answer continues to be the same - although they're good, there's still a lot to be done with Search. Contextual search is still being explored, and in terms of government and campaign information, most documents are not publicly or easily available to the search engines. With the goal of open government in mind, I decided to take a look at five relatively new search companies that recently launched sites, hoping that perhaps some of them could help make search of government and campaign data a little better, honing in on the FEC, OMB and more.
Here's a brief look at what they can do...
1) Cuil - (pronounced "cool") went live at the end of July. It has an interesting user interface (UI) that shows a portal-like visual of information found for each topic. The customizable view allows for varying screen sizes and it's easy to read.
2) Kosmix - aims to "organize the web". Born in 2006, it's still in Alpha, taking on verticals with its unique look. Well presented data and intuitive subdirectories, this site makes search seem less daunting.
3) SearchMe - "visual search" is in Beta. Around since 2005, they launched the first version in 2007 and went into private Beta last March. Now public, it requires a Flash package to be viewable - users can see images of pages at a glance. There's also an an iPhone application that searches visually. It can be slow, but pages are immediately expandable from within the search.
4) Deepdyve - (formerly Infovell) just released in November, this is a "deep search" contextual and document based search focused on the "Deep Web". They make agreements with and troll parts of the web not searchable by most others. The site currently requires signing up to use the service. (Disclaimer: my husband previously worked in Program Management there.)
5) Viewzi - recently advanced to public Beta, it provides a variety of data views, including the Power Grid View and the TechCrunch View. Loading data from Yahoo and Google, it essentially aims to make the data look better, with a variety of angles in which to read and process the information found.
For an initial test, I searched for Federal Election Commission because many journalists look at FEC filings to obtain information about candidates and campaign spending. The FEC site and opensecrets.org provide both easy interfaces for finding information, so this is a good comparison point. Cuil provided a variety of Advisory Opinions and a limited list of presidential candidates related to FEC. Kosmix, under "Related in the Kosmos", 8 categories were listed including Campaign Finance, Election, and the Bipartisan Campaign Reform Act of 2002. SearchMe brings up two visual stacks in a similar way to the Mac Leopard document viewing system - Campaigns & Elections and US Government. It brought up some interesting documents and moved quickly sifting them, but it took a little adjusting to the interface.
Deepdyve came up with a few interesting results and a variety of categories. When I refined the search, removing "life sciences", "physical sciences" and "clean tech" from the criteria, a Huffington Post article disappeared that was relevant. With Viewzi, the Google Timeline view of FEC data honed in on a Politico article about Obama escaping an FEC audit. The Simple Text View soon found a Report of Receipts and Disbursements for Hillary Clinton for President.
Next, I looked for anything related to the OMB (Office of Management and Budget). SearchMe brought up 4 cute little icons for US Government, Labor, Politicians and Blogs, but the pages presented under each option didn't vary much. Kosmix found "Our Miss Brooks" and Tomb Raider before the White House. Cuil came up with some good links. Viewzi went straight to relevant docs. Deepdyve located a few Annual Review papers.
Then I took it to another level, searching for "open government presidential directive technology infrastructure" and where Google came up with the Homeland Security Presidential Directive 7, Cuil pulled up a blog post from Tech Insider at GovernmentExecutive.com. Kosmix came up with the same directive 7 as Google, but provided several related categories worth searching. SearchMe located PediaView's definition of a presidential directive, and Viewzi found a report at businessofgovernment.org about "Strengthening Homeland Security." Deepdyve listed a Global Business Review article entitled "Transformation of an Emerging Economy to a Knowledge-Based Economy: Korean Case" and later a Public Works Management & Policy article about "Rail Security After 9/11."
Taking advantage of Deepdyve's document based search capabilities, I copied the Abstract of Presidential Directive 7 and pasted it into Deepdyve to see what would happen then. At that point, it brought up the rail security article first, followed by the Wikipedia entry on Critical Infrastructure Protection. Pasting the entire abstract into SearchMe brought up a Memorial Day open thread and a Canadian blog. Kosmix seemed confused, but it still found Directive 7. Cuil found a TSA document on Rail Transportation and a couple of GAO pages. Viewzi zoned in on last year's National Strategy for Homeland Security. Google went straight to Directive 7.
As to whether these new search engines have made any progress in contextual government search and whether any of them are helping us open government or improve media access, it's hopeful. If we could import a greater depth of government documents into a system like Deepdyve or SearchMe, particularly with sophisticated subcategorization, we might have some powerful tools. With all of the talk about the new administration reaching out and providing more open access, it should follow that more information will become publicly searchable. For anyone wanting to follow up on this, these sites are great places to start looking for ideas and suggestions to make to the new administration as they build out their technology and new policies.