Working with data sources in Hilo (Windows Store apps using JavaScript and HTML)

Article
10/13/2015

[This article is for Windows 8.x and Windows Phone 8.x developers writing Windows Runtime apps. If you’re developing for Windows 10, see the latest documentation]

From: Developing an end-to-end Windows Store app using JavaScript: Hilo

Previous page | Next page

The pages in Hilo use an in-memory data source to bind data to the Windows Library for JavaScriptListView control. While developing the month page for Hilo, we initially implemented a custom data source instead, but later changed the implementation to use an in-memory data source because our user experience (UX) requirements changed.

Download

After you download the code, see Getting started with Hilo for instructions.

You will learn

Tips on choosing the type of data source to implement in your app.
How to manage grouping in both an in-memory data source (WinJS.Binding.List) and a custom data source.
How to implement an in-memory data source and a custom data source.

Applies to

Windows Runtime for Windows 8
WinJS
JavaScript

Data binding in the month page

In this topic, we'll take a look at the ListView control in the month page as an example of data binding with different types of data sources. For more info about declarative binding for other controls used in Hilo and non-array data sources, see Using controls. For more info about binding, see Quickstart: binding data and styles. To meet UX requirements for the month page, we needed a flexible method to group and display a very large data set consisting of the images in local Pictures. The month page, shown in the following screenshot, displays images in month-based and year-based groups.

The ListView provides the flexibility we need to group and display data. We considered whether it would make sense to implement our own custom control, but the ListView is the best choice for us because it provides appearance and behavior consistent with Windows Store apps, built-in support for the cross-slide gesture, and it's performance optimized. For example, the ListView handles UI virtualization, recycling elements when they go out of view without destroying objects.

The month page includes a ListView control that's used for the main view. The second data-win-control element is used for the zoomed-out view (showing more months) that's provided by the SemanticZoom control.

Hilo\Hilo\month\month.html

<div class="semanticZoomContainer" data-win-control="WinJS.UI.SemanticZoom">

    <div id="monthgroup" data-win-control="WinJS.UI.ListView" data-win-options="{ 
        layout: {type: WinJS.UI.GridLayout},
        selectionMode: 'single'}">
    </div>

    <div id="yeargroup" data-win-control="Hilo.month.YearList">
    </div>

</div>

WinJS templates are used to format and display multiple instances of data. The month page uses several templates—three for the normal and snapped views (id="month*") and two for the zoomed-out view (id="year*") provided by the SemanticZoom control. (For more info about data binding and WinJS templates used in Hilo controls, see Using controls.) For the month page, we'll associate the ListView to the templates programmatically in JavaScript code. For more info about binding the data source to these properties, see Implementing group support with the in-memory data source

Here's the HTML code for the WinJS templates used for group items and group headers in the normal view.

Hilo\Hilo\month\month.html

<div id="monthItemTemplate" data-win-control="WinJS.Binding.Template">
    <div data-win-bind="style.backgroundImage: url.backgroundUrl; className: className"></div>
</div>

<div id="monthGroupHeaderTemplate" data-win-control="WinJS.Binding.Template">
    <a class="monthLink" href="#"><span data-win-bind="innerHTML: title"></span>&nbsp;(<span data-win-bind="innerText: count"></span>)</a>
</div>

<div id="monthSnappedTemplate" data-win-control="WinJS.Binding.Template">
    <span data-win-bind="innerHTML: title"></span>&nbsp;(<span data-win-bind="innerText: count"></span>)
    <div data-win-bind="style.backgroundImage: backgroundUrl;" class="thumbnail"></div>
</div>

In the preceding code, the data-win-bind attribute values specify declarative binding for DOM properties like the DIV element's backgroundImage (a URL), title, for the group header name (a month name), and count, for the number of items in the group.

Tip You can also use the data-win-options attribute, specifically the itemTemplate and groupHeaderTemplate properties, to declaratively associate the ListView control with a specific WinJS template.

[Top]

Choosing a data source for a page

The pages in Hilo use a WinJS.Binding.List for binding data to either the ListView or FlipView control.

For most pages, the WinJS.Binding.List is sufficient for our requirements, and is the only data source that type we implement. The WinJS.Binding.List provides a mechanism for creating groups from the primary data. The main limitation is that it's a synchronous data source, by which we mean that it is an in-memory data source (an array) and all its data must be available for display. Most Hilo pages display a somewhat limited file set. (The details page provides a set of files that belong to a single month, for example, and the hub page provides only six images in total.)

The month page uses a ListView to display images for all months in Pictures. For the month page, we wanted to ensure high performance for a very large data set, so we initially implemented a custom data source. Using a custom data source, we could obtain data for the ListView one page at a time. Later, our UX requirements for the month page changed to match those of the C++ version of Hilo. Hilo C++ displays only eight pictures per month in the month page. The ListView interface for the custom data source expected contiguous data, in order for it to support groups, so we decided to switch to a WinJS.Binding.List implementation. This UX change also diminished our need to support a very large data set, which made the decision easier. For more info about using a custom data source, see Other considerations: Custom data sources.

[Top]

Querying the file system

In the month page, we execute queries on Hilo's underlying data source, the file system, to obtain folders and files in Pictures. For more info about the query builder used to execute queries, see Using the query builder pattern.

To get images contained in a group, the month page uses the same query builder as other pages in Hilo. But to generate groups for the month page, Hilo uses a folder query instead of a file system query and passes a parameter (groupByMonth) to the Windows.Storage.Search.QueryOptions object. This query creates virtual folders based on the months read from each item's System.ItemDate property. In this code, folder contains a reference to the Windows.Storage.KnownFolders.picturesLibrary.

Hilo\Hilo\month\monthPresenter.js

_getMonthFoldersFor: function (folder) {
    var queryOptions = new search.QueryOptions(commonFolderQuery.groupByMonth);
    var query = folder.createFolderQueryWithOptions(queryOptions);
    return query.getFoldersAsync(0);
},

In the MonthPresenter, we will use the returned StorageFolder objects to create file queries for each month (not shown).

[Top]

Implementing group support with the in-memory data source

For ungrouped data, the use of the WinJS.Binding.List is fairly simple. After returning StorageFile objects from a file query, you need to pass in the array of items to a new instance of the list and assign its dataSource property to the ListViewitemDataSource property.

Hilo\Hilo\hub\listViewPresenter.js

setDataSource: function (items) {
    this.lv.itemDataSource = new WinJS.Binding.List(items).dataSource;
},

To use grouped data (pictures grouped into months), you need to assign the data to the ListView by using itemDataSource, but you also need to do some extra work beforehand. For each picture returned from the month page's file query, we set a groupKey property. This is used by the WinJS.Binding.List to group data internally. In the month page, we set groupKey to a unique value by using a numerical representation of the year that has the month appended, as shown in the following code. This value enables us to implement sorting as required by the List, and to sort items in the correct order.

Hilo\Hilo\month\monthPresenter.js

var buildViewModels = foldersWithImages.map(function (folder) {
    promise = promise.then(function () {
        return folder.query
        .getFilesAsync(0, maxImagesPerGroup)
        .then(function (files) {
            filesInFolder = files;
            // Since we filtered for zero count, we 
            // can assume that we have at least one file.

            return files.getAt(0).properties.retrievePropertiesAsync([itemDateProperty]);
        })
        .then(function (retrieved) {
            var date = retrieved[itemDateProperty];
            var groupKey = (date.getFullYear() * 100) + (date.getMonth());
            var firstImage;

            filesInFolder.forEach(function (file, index) {
                var image = new Hilo.Picture(file);
                image.groupKey = groupKey;
                self.displayedImages.push(image);

                if (index === 0) {
                    firstImage = image;
                }
            });

Prior to setting the group key, we also wrap each StorageFile in a Hilo.Picture object to make it bindable to the ListView. (They are not bindable because Windows Runtime objects such as StorageFile are not mutable.) For more info, see Using the query builder pattern.

To implement groups by using the WinJS.Binding.List, we create a new instance of the List and call the createGrouped method. For grouping to work correctly, we pass createGrouped three functions:

groupKey, to obtain the group key for each picture.
groupData, to obtain an array of group data for each picture.
groupSort, to implement sorting of the data.

This code shows the implementation of these functions, the creation of the List object, and the call to createGrouped.

Hilo\Hilo\month\monthPresenter.js

_createDataSources: function (monthGroups) {
    var self = this;

    function groupKey(item) {
        return item.groupKey;
    }

    function groupData(item) {
        return self.groupsByKey[item.groupKey];
    }

    function groupSort(left, right) {
        return right - left;
    }

    var imageList = new WinJS.Binding.List(this.displayedImages).createGrouped(groupKey, groupData, groupSort);

In the preceding code, groupData calls the presenter's groupsByKey function to get the group information required by the List. The group information has already been assigned to the month page presenter's groupsByKey property. Here's the code in which we set the group information properties and assign it to groupsByKey. (This code runs once for each month's file query, before creating the List.)

Hilo\Hilo\month\monthPresenter.js

var monthGroupViewModel = {
    itemDate: date,
    name: firstImage.name,
    backgroundUrl: firstImage.src.backgroundUrl,
    title: folder.groupKey,
    sortOrder: groupKey,
    count: folder.count,
    firstItemIndexHint: firstItemIndexHint,
    groupKey: date.getFullYear().toString()
};

firstItemIndexHint += filesInFolder.size;
self.groupsByKey[groupKey] = monthGroupViewModel;
groups.push(monthGroupViewModel);

If you don't call the createSorted function of the List in your code, the List sorts data by passing the groupKey values to your implementation of the groupSort function. (See the preceding code.) In Hilo, this means that the month/year numerical value is used for sorting data.

It is also worth noting that firstItemIndexHint will be used to specify the raw index for the first item in each month group. This value is used to build a query specific to a month when a picture is selected (not shown), which causes the app to navigate to the details page.

[Top]

Other considerations: Using a custom data source

This section provides details about the working implementation of a custom data source that we initially created for Hilo. For Hilo source code that implements the custom data source, see this version of Hilo.

In this section, we'll describe the following:

Considerations in choosing a custom data source
Implementing a data adapter for a custom data source
Binding the ListView to a custom data source

[Top]

Considerations in choosing a custom data source

If we didn't need to match the Hilo C++ UX, which shows only eight images for each month in the month page, we would have used a custom data source in Hilo. (The ListView interface for the custom data source expected contiguous data to support groups, so we decided to switch to a WinJS.Binding.List implementation, which uses an in-memory array of data.) A custom data source would have provided data virtualization, enabling us to display images one page at a time and avoid any delay resulting from the use of WinJS.Binding.List.

For data sources other than the in-memory list, choices include a StorageDataSource object or a custom data source. A StorageDataSource includes built-in support for the Windows file system, but it doesn't support grouping of items, so we chose to implement a custom data source.

For grouped data, two data sources are required—one for the groups and one for the items.

A custom data source must implement the IListDataAdapter interface. Both data sources in the month page use data adapters that implemented this interface. When you implement this interface, the ListView infrastructure can make item requests at run time, and the data adapter responds by returning the requested information or by providing information stored in a cache. If images aren't loaded yet, the ListView displays placeholder images (which are specified in month.css).

For more info about the ListView and data sources, see Using a ListView. For more info about custom data sources, and an example that uses a web-based data source, see How to create a custom data source.

[Top]

Implementing a data adapter for a custom data source

A custom data source must implement the IListDataAdapter interface. In Hilo, the DataAdapter class implemented IListDataAdapter. Most of the challenges of implementing a custom data source had to do with basing the data source on the file system, and in particular, creating groups with a correspondence to image files only. Objects returned from the file system—StorageFolder and StorageFile objects—aren't in a format that's bindable to the ListView, so you have to do some work to generate usable data. For the groups and items required in the month page, we needed to provide data in the special formats required by the IListDataAdapter interface.

Important The final version of Hilo did not implement a custom data source. For Hilo code that implements the custom data source, see this version of Hilo.

For all pages in Hilo, we also wrap returned images in Hilo.Picture objects, which are implemented in Picture.js. For more info, see Using the query builder pattern.

The key requirements to implement IListDataAdapter include adding implementations for getCount, itemsFromIndex, and itemsFromKey.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    var DataAdapter = WinJS.Class.define(function (queryBuilder, folder, getMonthYearFrom) {
        // . . .
        };

    }, {

        itemsFromIndex: function (requestIndex, countBefore, countAfter) {
        // . . .
        },

        getCount: function () {
        // . . .
        },

        itemsFromKey: function (key, countBefore, countAfter) {
        // . . .
        },

The getCount method returns the total count of all items currently in the data source. This number must be either be correct or return null, or the ListView might not work correctly, and it might not indicate a reason for failure.

During our investigation period, we believed that implementing itemsFromKey was optional, but at run time the ListView infrastructure required an internal object derived from the itemsFromKey implementation, so we added a simple implementation of itemsFromKey to the group data adapter.

Important A similar data adapter for items is implemented in members.js. Because the data adapter for items was easier to implement and didn't require extra code to handle the grouping of data, we focus on the groups in this topic. The data adapter for items didn't require an implementation of itemsFromKey either.

At run time, when the ListView needs a set of items (groups, in this case) to populate the ListView, it calls itemsFromIndex, passing it a request for a specific group index value along with suggested values for the number of groups to return before and after the requested index value.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

   itemsFromIndex: function (requestIndex, countBefore, countAfter) {
        var start = Math.max(0, requestIndex - countBefore),
            count = 1 + countBefore + countAfter,
            offset = requestIndex - start;

        // Check cache for group info. If not complete, 
        // get new groups (call fetch).

        // . . . 

        var buildResult = this.buildResult.bind(this, offset, start);

        return collectGroups.then(buildResult);
    },

The return value for itemsFromIndex needs to be an object that implements the IFetchResult interface. In Hilo, we create an object with the following properties to fulfill the interface requirements:

items, which must be an array of items (groups) that implement the IItems interface.
offset, which stores the index for the requested group relative to the items array.
absoluteIndex, which stores the index of the requested group relative to the custom data source.
totalCount, which stores the total number of groups in the items array.

To get these values, we calculate the offset and absoluteIndex values by using requestIndex, countBefore, and countAfter, which are the requested values passed in by the ListView. (See the preceding code.) We retrieve the totalCount from the data adapter's cached value.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    buildResult: function (offset, absoluteIndex, items) {
        // Package the data in a format that the consuming 
        // control (a list view) can process.
        var result = {
            items: items,
            offset: offset,
            absoluteIndex: absoluteIndex
        };

        if (this.totalCount) {
            result.totalCount = this.totalCount;
        }

        return WinJS.Promise.as(result);
    },

To get the array of requested groups (the items property), we make file system queries by calling functions like getFoldersAsync, which returns an array of StorageFolder objects. We call getFoldersAsync in the fetch function (called from itemsFromIndex), and then use the JavaScript array mapping function to convert the StorageFolder objects to usable data.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    return this.query
        .getFoldersAsync(start, count)
        .then(function (folders) {
            return WinJS.Promise.join(folders.map(convert));
        })

In convert, the callback function for array mapping, we make additional file system queries by using the passed in StorageFolder object, and then pass the query results to buildMonthGroupResult in a joined promise.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    var query = this.queryBuilder.build(folder);

    var getCount = query.fileQuery.getItemCountAsync();

    var getFirstFileItemDate = query.fileQuery
        .getFilesAsync(0, 1)
        .then(retrieveFirstItemDate);

    return WinJS.Promise
        .join([getCount, getFirstFileItemDate])
        .then(this.builldMonthGroupResult.bind(this));

In buildMonthGroupResult, we create a group object that implements the IItems interface, and that can be set as the items property value of the IFetchResult. To implement the IItems interface, we use the query results to set group properties like groupKey, data, index, and key. For example, the query results from getFilesAsync(0,1) are used to set group properties based on the item date of the first file in the group.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    builldMonthGroupResult: function (values) {

        var count = values[0],
                    firstItemDate = values[1];

        var monthYear = this.getMonthYearFrom(firstItemDate);

        var result = {
            key: monthYear,
            firstItemIndexHint: null, // We need to set this later.
            data: {
                title: count ? monthYear : 'invalid files',
                count: count
            },
            groupKey: count ?  monthYear.split(' ')[1] : 'invalid files'
        };

        return result;
    }

Important The key property must be unique or the ListView may appear to fail without providing an error message.

The data properties, title and count, are declaratively bound to the UI in the WinJS template for the group header defined in month.html.

Important This function also creates properties like firstItemIndexHint, which is used to specify the first index value in each group. Its value is set later.

For more info on the use of promises in Hilo, see Async programming patterns and tips.

[Top]

Binding the ListView to a custom data source

When the month page is loaded, the ready function for the page control is called, in the same way that it's called for other Hilo pages. In the ready function, two IListDataAdapter objects are created:

A data adapter object that represents the groups, which is instantiated by calling Hilo.month.Groups.
A data adapter that represents the items, which is instantiated by calling Hilo.month.Items.

In both cases, we pass the query builder to the IListDataAdapter along with several other parameters.

Important The final version of Hilo didn't implement a custom data source. For Hilo code that implements the custom data source, see this version of Hilo.

Hilo\Hilo\month\month.js (in version of Hilo with a custom data source)

    ready: function (element, options) {

        // I18N resource binding for this page.
        WinJS.Resources.processAll();

        this.queryBuilder = new Hilo.ImageQueryBuilder();

        // First, set up the various data adapters needed.
        this.monthGroups = new Hilo.month.Groups(this.queryBuilder, this.targetFolder,
            this.getMonthYearFrom);
        this.monthGroupMembers = new Hilo.month.Members(this.queryBuilder, 
            this.targetFolder, this.getMonthYearFrom);

        var yearGroupMembers = this._setYearGroupDataAdapter(this.monthGroups);
        var yearGroups = yearGroupMembers.groups;

        // Then provide the adapters to the list view controls.
        this._setupMonthGroupListView(this.monthGroups, this.monthGroupMembers);
        this._setupYearGroupListView(yearGroups, yearGroupMembers);
    },

Any data source that binds to a ListView, including a WinJS.Binding.List, must implement IListDataSource. To bind to a ListView (or a FlipView) by using a custom data source, the data source you create must derive from VirtualizedDataSource, which implements IListDataSource. The VirtualizedDataSource is the base class for an IListDataAdapter.

The DataAdapter class implements IListDataAdapter. In this code, called from the ready function, we create the group data adapter—an object derived from VirtualizedDataSource—and pass a new DataAdapter object to the constructor for the newly derived class. To instantiate the new object correctly, we also need to pass the DataAdapter object to _baseDataSourceConstructor, the base class constructor for a VirtualizedDataSource.

Hilo\Hilo\month\groups.js (in version of Hilo with a custom data source)

    var Groups = WinJS.Class.derive(WinJS.UI.VirtualizedDataSource, 
        function (queryBuilder, folder, getMonthYearFrom) {
        this.adapter = new DataAdapter(queryBuilder, folder, getMonthYearFrom);
        this._baseDataSourceConstructor(this.adapter);
    }, {
        getFirstIndexForGroup: function (groupKey) {
            var groupIndex = this.adapter.cache.byKey[groupKey];
            var group = this.adapter.cache.byIndex[groupIndex];
            return group.firstItemIndexHint;
        }
    });

After the data adapters for groups and members are created, data is returned asynchronously from the file system upon request by the ListView. For more info, see Implementing a data adapter for a custom data source.

In the preceding code, the ready function calls _setupMonthGroupListView and _setupYearGroupListView to bind the data sources to the ListView In the following code, the monthgroupListView control declared in month.html is retrieved, and then the groupDataSource and itemDataSource properties are used to bind the groups and members from the Hilo data adapters to the ListView.

Hilo\Hilo\month\month.js (in version of Hilo with a custom data source)

    _setupMonthGroupListView: function (groups, members) {
        var listview = document.querySelector("#monthgroup").winControl;
        listview.groupDataSource = groups;
        listview.itemDataSource = members;
        listview.addEventListener("iteminvoked", this._imageInvoked.bind(this));
    },

[Top]

Working with data sources in Hilo (Windows Store apps using JavaScript and HTML)

Download

You will learn

Applies to

Data binding in the month page

Choosing a data source for a page

Querying the file system

Implementing group support with the in-memory data source

Other considerations: Using a custom data source

Considerations in choosing a custom data source

Implementing a data adapter for a custom data source

Binding the ListView to a custom data source

Additional resources