• Article

December 2013

Volume 28 Number 12

Windows Store Apps - Freedom of Information Act Data at Your Fingertips

By Vishwas Lele

This article is from MSDN Magazine's special coverage of application development in the government sector.
Read more articles like this in the Special Government Issue of MSDN Magazine.

The Freedom of Information Act (FOIA) gives U.S. citizens access to government data to help educate themselves and make more informed decisions. The process of getting this data, however, can be complex and daunting. In this article, I’ll walk through the building of a Windows Store app, called MyFOIA, that lets users more easily get this information, share it with others and even get automatic updates on FOIA requests.

What Is the FOIA?

The FOIA grants citizens the right to request information from any government agency. The agency is then required to provide the citizen with any pertinent, non-classified information. This law is designed to increase governmental transparency. However, the request process doesn’t allow for easy access to an agency’s information.

Most agencies offer frequently requested information on their Web sites. For information that doesn’t appear there, a person must make an FOIA request. This process often requires the citizen to submit a written request by postal mail or e-mail, outlining in as much detail as possible the information the person wants and the desired format in which it should be provided. The request must be compliant with certain specific format guidelines, which vary from agency to agency. The typical return time on an FOIA request can be at least one month. These factors raise the need for a tool that can improve access to this type of government information.

In a system of government designed by the people, for the people, the FOIA helps citizens access information that previously would’ve been available only to federal agencies.

The law has helped citizens uncover results from unpublished studies, previously withheld videos and unreported breaches of regulations, all of which wouldn’t have been otherwise available to the public. See “The FOIA in Action” in this article for details about some real examples.

Challenges of the FOIA

The FOIA can be a powerful tool for analyzing and extracting useful conclusions from government data. However, the process is long and complicated. Under the current system, the citizen must first figure out—as specifically as possible—what he’s looking for. Sometimes the data a citizen wants is locked in an agency-specific portal.

To combat this problem, there have been some multi-agency databases created, such as FOIAonline (1.usa.gov/O0s4zu). Here, the citizen has two options: search the frequently requested FOIA data or make an official FOIA request. The official FOIA request requires divulging personal information to create an account, in addition to submitting a detailed letter of description outlining the request.

Depending on the agency, the requester may also be required to submit a fee. This process can be arduous and relies on the citizen actively seeking the information from the agency by means of a request, rather than having the data widely available at the person’s fingertips. A user can’t receive notification upon any updates to the data in the system, and there’s no easy way for people to share or discuss this information by means of social media.

Debrief: The Freedom of Information Act (FOIA)

Transparency of government data plays a vital role in keeping U.S. citizens informed of the activities of government agencies and involved in the governing process. The FOIA makes this data available, but requesting and receiving this information can be a daunting and cumbersome experience. This article details the development of a Windows Store app that improves on the current processes for working with FOIA requests.

 

IT Brief:

This article can serve as an example for government agencies of how to improve the process of fulfilling the FOIA mandate. As an alternative to developing completely new solutions, it shows how emerging technologies can augment and improve current systems. Important considerations include the following:

  • Government agencies are required to comply with FOIA requests.
  • Current processes for fulfilling requests vary from agency to agency.
  • Filing and tracking FOIA requests can be difficult for citizens, but the process can be standardized and improved.
  • Using new technology such as Windows Store apps can help standardize and enhance the process.

Dev Brief:

Developers might not always have the time or resources to completely rebuild current systems. This article shows how to use modern technologies, such as the new and improved WebView control in Windows 8.1, to interact with and improve existing systems. In this case, an app works with an existing FOIA-related Web site and adds the following capabilities:

  • Users can be notified if there’s a change to an FOIA request in which they were interested and following.
  • Users can easily share FOIA-related data over social media.
  • Users can interact with the app smoothly via a variety of input sources, including touch, pen, mouse and keyboard.

More Information:

Enter the MyFOIA App

I decided it would be useful to build a Windows Store app to make FOIA-related information more accessible. Users would be able to easily access FOIA data by searching for existing FOIA requests and creating new ones. Furthermore, a user would be notified of any updates to an FOIA request she wanted to follow. Finally, the app would make it easy to share interesting tidbits related to FOIA data over social media, thereby encouraging others to get involved.

I’ll now briefly review the functionality provided by the app.

Home Page Figure 1 depicts the Home page of the MyFOIA app. Users can search for requests, make a new request (as a guest) and generate reports. The app bar also allows for navigation controls and enables users to tweet selected text from within the app.

The MyFOIA App Home Page
Figure 1 The MyFOIA App Home Page

Search Page Figure 2 depicts the Search page of the MyFOIA app. Users can search for requests, appeals and records released in response to FOIA requests. In this example, the user is searching through the U.S. Environmental Protection Agency’s (EPA) FOIA records for the search term “CO2.”

The MyFOIA App Search Page
Figure 2 The MyFOIA App Search Page

Search Results Figure 3 depicts the search results page. Results of the search placed earlier (in Figure 2) are displayed on the screen. Notice the column at the right with the heading of Notify. Clicking on one or more checkboxes allows users to receive notifications if there’s an update related to the request.

The MyFOIA App Search Results Page
Figure 3 The MyFOIA App Search Results Page

Select and Share Page Figure 4 depicts the “select and share” page that lets users select interesting tidbits from responses and share them over Twitter. Notice the browser window with the Twitter page alongside the MyFOIA app.

The MyFOIA App Select and Share Page
Figure 4 The MyFOIA App Select and Share Page

High-Level Architecture

Figure 5 depicts the high-level architecture of the MyFOIA app, which was built using JavaScript and HTML. It relies on the Windows 8.1 WebView control to host content from the FOIAonline site, thereby making available the core FOIAonline functionality (create, search and report on responses). The app also extends functionality provided by the FOIAonline Web site by “injecting” JavaScript into the WebView-hosted content. For example, it lets users track FOIA responses. If there’s an update to an FOIA response, all devices that have registered to receive updates will receive push notifications. The MyFOIA app also extends the look and feel of the FOIAonline Web site to match a native Windows Store app. This includes enabling “pinch and zoom,” relying on an app bar for navigation, applying necessary styling to make the app touch-friendly and sharing text with other Windows Store apps.

The High-Level Architecture of the MyFOIA App
Figure 5 The High-Level Architecture of the MyFOIA App

The other key piece of functionality offered by the MyFOIA app is the notification capability. Users can register to receive notifications whenever a given FOIA response is updated. This notification capability is built using Windows Azure Mobile Services (WAMS), which enables back-end capabilities for apps, including: 

  • Simple provisioning and management of tables where apps can store data.
  • Integration with the notification hub to deliver push notifications.
  • Client libraries for various devices, including JavaScript libraries.
  • The ability to add script-based server logic, including scheduler capabilities.

Each device, at app activation time, requests a push notification channel from the Notification Client Platform. In turn, the Notification Client Platform asks Windows Push Notification Services (WNS) to create a notification channel and return it to the device. Subsequently, when the user enables tracking on a given FOIA response, the MyFOIA app registers the device with the Windows Azure notification hub. In the background, the WAMS scheduler periodically looks for updates to all registered FOIA responses. If the scheduler discovers an update, a push notification is sent to all devices via the notification hub.

Implementation Details

The first challenge in developing the app was that the FOIAonline Web site doesn’t currently support an API for app developers. Fortunately, Microsoft has released an updated version of the WebView control as part of Windows 8.1 that lets me host an FOIAonline Web page within a Windows Store app built with native HTML and JavaScript (the earlier version of the WebView control in Windows 8 was accessible only from apps built with XAML and C#).

You’re probably thinking this uses some sort of an iframe-based approach. Not quite. The WebView control offers several advantages over an iframe-based approach. Also, some sites simply disallow their content to be loaded within an iframe.

In simple terms, the new WebView control allows an HTML app to host HTML content. Consider the following HTML and JavaScript snippets used to create the WebView control:

<x-ms-webview id="foiaWebview"></x-ms-webview>
var foiaWebview = document.createElement("x-ms-webview");

Note that x-ms-webview follows vendor-specific extension syntax, because WebView is a Microsoft-specific control. Once you’ve created the WebView control, you can load the HTML content as a new document using the navigateToString method:

x-ms-webview.navigateToString(stringHTMLContent);

So far the functionality looks similar to that provided by an iframe, but there are significant differences:

  • The WebView control is integrated into the display tree along with other controls on the HTML page, so you can apply styling just as with other controls on the page. This means you can also overlay other controls on top of the WebView control.
  • The WebView control offers a set of navigation events that provides the app with an insight into the loading of content. Apps can register for events such as MSWebViewNavigation­Starting, MSWebViewContent­-Loading, MSWebViewContent­Loaded and MSWebViewNavigation­Completed, and take appropriate action. In addition, the WebView control also periodically (every 500 ms) fires a LongRunningScriptDetected event that lets the app halt a potentially errant script.
  • The WebView control can take advantage of Internet Explorer SmartScreen filtering to block “phishing” attacks. The app can be notified of malicious content and navigate away from the page.
  • The WebView control comes with built-in functionality that’s commonly needed by apps hosting Web content. This includes checking for the existence of “next” and “previous” links and navigating accordingly, capturing screenshots of the Web content being displayed, and remembering a user’s selection.
  • For functionality that isn’t built-in, the WebView control provides a generic scheme for the app to communicate with the WebView control via the InvokeScriptAsync method and ScriptNotify event. As the name suggests, InvokeScriptNotify provides—as an asynchronous action—the ability to exe­cute a script from within the currently loaded HTML inside the WebView control. Analogously, the HTML page within a WebView can raise a ScriptNotify event within the app. As you can imagine, a number of security checks are imposed in order to prevent externally hosted, malicious Web content from hijacking the app. For example, only the Web sites registered in the app package manifest are allowed to raise events within an app.
  • Functionality such as pinch and zoom and phone-number detection works out of the box with the WebView control. Although you can certainly achieve similar behavior using iframes, it would require a bunch of custom Java­Script and CSS.

Now that you’re armed with knowledge of how the WebView control works, I’ll discuss how the MyFOIA app uses the WebView control. The start page of the MyFOIA app is default.html, as defined in the app package manifest. The default.html page simply hosts the WebView control, along with the WinJS.UI.AppBar control. When the app is activated, the app.onactivated event (defined in default.js) gets invoked. Inside its event handler, you register for the events discussed previously, including MSWebViewNavigationStarting, MSWebViewDOMContentLoaded and MSWebViewScriptNotify. Finally, you navigate to the foiaHome.html page. This is a page bundled with the app package, which is why you’ll need an ms-appx-web prefixed URL to access it, like this:

var homePageUri = "ms-appx-web:///foiaHome.html";

If you look at the contents of this page, it looks similar to the homepage of the FOIAonline site. I bundled this page as part of the app for two reasons. First, I needed a fast, responsive, touch-friendly home screen that stretched to fit the entire on-screen real estate. Second, I wanted to limit the app to the functionality available to guest users only. Bundling a copy of the homepage allowed me to remove the functionality related to registered users. Specifically, the MyFOIA app has no functionality to solicit user credentials.

Figure 6 depicts the homepage of the FOIAonline site. Compare and contrast it to the MyFOIA app Home page (Figure 1) and you’ll notice that several items (annotated in red on Figure 6) have been removed in order to provide a more fluid UI experience.

The FOIAonline Home Page
Figure 6 The FOIAonline Home Page

The next challenge I ran into was enabling some sort of push notification capability that notifies users of updates to an FOIA response in which they were interested. However, no such push notification capability exists within the FOIAonline Web site. Fortunately, the InvokeScriptAsync and ScriptNotify methods discussed earlier let me “inject” the push notification capability within the existing Web page. Refer to Figure 3 and you’ll notice the additional Notify column. This column doesn’t exist on the FOIAonline Web site.

I’ll explain how I was able to add this additional column. Once DOM content is loaded inside the WebView control, it will fire the foiaWebview_onDOMContentLoaded event (recall that I registered for it inside the app-activated event). This lets the MyFOIA app modify the behavior of the hosted content by injecting a custom script that adds the Notify column to the existing HTML. This code creates a dependence on the underlying Web site. As a result, the MyFOIA app can break if the underlying Web site is modified. In the absence of an API, the WebView control offers the only option to extend the functionality offered by the Web site.

In Figure 7, you can see that along with adding the Notify column, I pass in the “eval” script function, along with a script argument, to the webviewControl.invokeScriptAsync method.

Figure 7 Script Function to be Executed from the Currently Loaded HTML

script += "function notifyClick(checkboxControl) {";
script += "var notifyData = '';";
script += "if (checkboxControl.checked) {";
script += "    notifyData = 'add';";
script += "}";
script += "else {";
script += "    notifyData = 'delete';";
script += "}";
script += "window.external.notify(notifyData + '|' + ";
script += " $(checkboxControl).attr('data-number') + '|' + ";
script += " $(checkboxControl).attr('data-uri'));";
script += "}";
script += "var header = $('#curElem thead tr');";
script += "$(header[0]).find('th:last').after('";
script += " <th class=\"detail\">";
script += " <a href=\"#\" onclick=\"return false\">Notify</a>";
script += " </th>');";
script += "var rows = $('#curElem tbody tr');";
script += "for (var i = 0; i < rows.length; i++) {";
script += "    var reqNumber = $(rows[i]).find('td:first a').text();";
script += "    var reqUri = $(rows[i]).find('td:first a').attr('href');";
script +=   "reqUri = 'https://foiaonline.regulations.gov/foia/action/public/view/' +";
script +=   "(reqUri.indexOf('request') != -1 ? \"request?\" : \"record?\") + ";
script += "     reqUri.substring(reqUri.indexOf('objectId'));";
script += "    $(rows[i]).find('td:last').after('";
script += "     <td><input type=\"checkbox\" class=\"notifyCheckbox\" ";
script += "     name=\"notify\" value=\"notify\" ";
script += "     data-number=\"' + reqNumber + '\" data-uri=\"' + reqUri + '\" ";
script += "     onclick=\"notifyClick(this);\" /></td>');";
script += "}";
var scriptOperation =
   webviewControl.invokeScriptAsync("eval", new Array(script));

Upon successfully completing this, the custom script is executed within the loaded HTML page. Note that within the script snippet, passed in as an argument, I include a call to method window.external.notify. So when the user selects one of the checkboxes, an foiaWebview_onScriptNotify event is fired within the MyFOIA app. Figure 8 depicts how this event is handled by the app. I invoke the WAMS method called getTable to obtain a reference to a specific table. Using the table object, I add a new record con­tain­ing the number of the FOIA request being tracked and the Uri of the request.

Figure 8 The onScriptNotify Event Handler

function foiaWebview_onScriptNotify(eventArgs) {
  msgControl.innerText = "Updating Request...";
  var scriptNotifyDataArr = eventArgs.value.split("|");
  var operation = scriptNotifyDataArr[0];
  var reqNumber = scriptNotifyDataArr[1];
  var reqUri = scriptNotifyDataArr[2];
  var requestTable = foiaMobileServiceClient.getTable("Request");
  var tablePromise;
  if (operation == "add") {
    requestTable.insert({
      DeviceId: deviceId,
      TrackingNumber: reqNumber,
      RequestUri: reqUri
  }).done(function () {
    updateRegistration();
  });
  }
  // Code elided for clarity
  }
}
function updateRegistration() {
  var channelOperation =
     pushNotifications.PushNotificationChannelManager.
    createPushNotificationChannelForApplicationAsync();
  channelOperation.then(function (newChannel) {
    channel = newChannel.uri;
    return requestTable.where({ DeviceId: deviceId }).read();
  }).then(function (requests) {
  if (requests.length > 0) {
    var trackingNumArray = getTrackingNumbers(requests);
    return hub.registerApplicationAsync(channel, trackingNumArray);
  }
  // Code elided for clarity

Once the record is successfully inserted, I also register the channel and tracking numbers with the notification hub. The newly inserted record is then picked up by a custom script fired periodically by the WAMS scheduler. The custom script checks for updates to the FOIA request using the Uri column. If there are any updates, it sends out notifications using the notification hub. The benefit of using the notification hub is it scales easily to a large number of recipients without the need for re-architecting the MyFOIA app. Also, the notifications are based on tags. Tags are a way to register MyFOIA app user preferences. In concrete terms, tags in the MyFOIA app are tracking numbers of the FOIA requests that a user is interested in tagging. You can register one or more tags. The notification hub then uses tags to route the notifications.

The final challenge was related to adding social media capabilities to the app. I thought it would be interesting to share “tidbits” from FOIA requests and responses over social media tools such as Twitter. Once again, the FOIAonline Web site doesn’t currently support such a capability. The challenge in implementing such a capability is that users would be selecting text within the hosted HTML page. How does the app capture the selected text? Fortunately, as shown in Figure 9, the WebView control offers a method called captureSelectedContentToDataPackageAsync that captures the selected text and passes it to the loadTwitter method. The loadTwitter method in turn uses the Windows.System.Launcher.launchUriAsync method to launch the default app associated with the URI. It’s also interesting to note the use of the Windows.UI.ViewManagement.ViewSizePreference option, introduced in Windows 8.1, to define app view size preference. This lets me launch the browser app alongside the MyFOIA app, as shown in Figure 4.

Figure 9 Capturing Selected Text and Sharing It with Twitter

function mainAppBar_onTweet(eventArgs) {
  msgControl.innerText = "Loading...";
  var captureOperation =
   webviewControl.captureSelectedContentToDataPackageAsync();
  captureOperation.oncomplete = function (completeEvent) {
    var res = completeEvent.target.result;
    if (res) {
      var dataPackage = res.getView();
      dataPackage.getTextAsync().done(function (capturedText) {
        loadTwitter(capturedText);
      });
    }
    else {
      loadTwitter("");
    }
  };
  captureOperation.start();
}
function loadTwitter(tweet) {
  var url = new Windows.Foundation.Uri(
  "https://twitter.com/share?url=https://foiaonline.regulations. "+
  "gov/foia/action/public/home&text=" + tweet);
  var options = new Windows.System.LauncherOptions();
  options.desiredRemainingView =
     Windows.UI.ViewManagement.ViewSizePreference.useMore;
    Windows.System.Launcher.launchUriAsync(url, options).done(
      function (data) {msgControl.innerText = "";
  });
}

The FOIA in Action

The following news articles were made possible by use of the FOIA:

  • “A Breach of Truth,”Chattanooga Times Free Press(Tennessee),March 4, 2006 This article describes how the FOIA allowed the release of a video that shows a briefing conducted for President George W. Bush by Michael Brown, director of the Federal Emergency Management Agency (FEMA). In the video, experts express fears that a hurricane could flood New Orleans with a resulting high death toll, to which President Bush responds “we are fully prepared.”
  • “Feds fault Chiron for lax cleanup of flu shot plant,”San Francisco Chronicle, June 21, 2006 Chiron Corp., a British pharmaceutical company, owned a Liverpool plant that produced 50 percent of the influenza vaccine used by the United States. This article describes how, in 2005, information released under the FOIA led to the discovery that this Chiron plant didn’t meet Food and Drug Administration (FDA) regulations. In 2004, the FDA recalled and destroyed the plant’s entire production run, leading to a flu vaccine shortage in the winter of that year. The information disclosed showed that Chiron’s vaccines were eventually cleared by October 2005, which led to concern for U.S. citizens awaiting the vaccine.
  • “On Range, deadly illness went unreported; Mesotheliomastrikes years after victims’ exposure to asbestos,”Star Tribune(Minneapolis, Minn.), Aug. 21, 2005 This article describes how the Mine Safety and Health Administration (MSHA) requested records under the FOIA that exposed a loophole in report requirements that let LTV Steel Mining Co. forego reporting a trend of mesothelioma and other debilitating asbestos-related illnesses among workers in its Minnesota taconite mines dating from 1980. Because mesothelioma usually doesn’t appear for more than 20 years after exposure to asbestos, LTV didn’t report illnesses and deaths among its retirees, (it is required to do so for its active workers). Because of this loophole, there was no action taken to improve safety of other workers at the mine. The failure to report lung disease cases among mine workers was discovered from examination of the documents requested, after reporters spoke with families of affected workers in the Iron Range region. In addition, the MSHA also discovered that the maximum penalty for companies that failed to report an illness was $60.

The exposure of breaches of regulations, suppressed health studies and other information publicized by these articles demonstrates the importance the FOIA plays in keeping citizens informed about the data obtained by federal agencies. By granting the capability for citizens to make requests of any and all information from government agencies, the FOIA provides citizens with the same access to government information.

Stay in the Know

The FOIA is a law that gives citizens the right to access information from the federal government. It’s often described as the law that keeps citizens in the know about their government. The MyFOIA Windows Store app is designed to make the FOIA data easily accessible. The MyFOIA app is built using WebView control and WAMS. The WebView control makes it possible to not only take advantage of the existing FOIAonline Web site but also enhance it. WAMS makes it possible for the users to be notified of updates to an FOIA response in which they're interested.

The MyFOIA app is a work in progress, but I’ve posted the code at github.com/AppliedIS/Foia for you to review.

Finally, I’d like to thank Pamela Steger for help with this article and Sajad Deyargaroo for help with the development of the MyFOIA app.

Vishwas Lele is the CTO at Applied Information Sciences Inc. He is responsible for assisting organizations in envisioning, designing and implementing enterprise solutions. Lele also serves as the Microsoft regional director for the Washington, D.C., area and is a Windows Azure MVP. You can reach him on Twitter at twitter.com/vlele.

Thanks to the following Microsoft technical experts for reviewing this article: Kraig Brockschmidt, Kevin Hill and Jake Sabulsky.
Kraig Brockschmidt has worked with Microsoft since 1988, focusing on helping developers through writing, education, public speaking and direct engagement. He’s a senior program manager on the Windows Ecosystem team working with key partners on building Windows Store apps and bringing knowledge gained in that experience to the wider developer community. His most recent book is "Programming Windows 8 Apps in HTML, CSS, and JavaScript" (a free ebook from Microsoft Press).

Kevin Hill (khill@microsoft.com)

Jake Sabulsky is a program manager on the Application Experience team in Windows. He has worked in Windows since 2009. Jake’s two areas of focus during that time have been building the Web application platform and enhancing the app-to-app communication story for Windows Store apps.