Overview: ADO.NET Data Services

Microsoft Data Platform Development Technical Article

Writers: Mike Flasko
Microsoft Corporation
Published: August 2008
Applies To:
            Visual Studio 2008 SP1
           .NET Framework 3.5 SP1
           ADO.NET Data Services

Summary: The ADO.NET Data Services framework consists of a combination of patterns and libraries that enable the creation and consumption of data services for the web. This article will discuss the key elements of the ADO.NET Data Services framework at a high level and the motivation behind its design.

>>Introduction

>Driven by the need to provide richer user experiences while maintaining the ease of deployment that is characteristic of web content, the way web applications are architected is changing dramatically.

The technological shift towards AJAX-based frameworks and tools, and the trend for building Rich Interactive Applications based on application environments such as Microsoft Silverlight 2 and Adobe Flash, changes how presentation, behavior and data are delivered to client web browsers.

In traditional web applications, a server-side component would render HTML content that consisted of presentation aspects (fonts, colors, and layout), the "data" or content itself, and perhaps client-side code (typically JavaScript) to drive interaction. All of this was delivered to the web browser through a single mechanism and every interaction would require going back to the server "with the whole page" for processing. Finally, another HTML page would be rendered as the response.

One key trait of the new architectures for web applications is that presentation and data are no longer embedded in the same container and delivered through the same channel. AJAX-based web sites serve pages containing presentation and behavior, and then the JavaScript code that represents the page behavior turns back and fetches data separately using XMLHTTP. Silverlight and Flash applications remove the option of a server-side rendering process that mixes data and code; code to drive the presentation aspects is pre-compiled and deployed as a single file in the web server. After reaching the client web browser the code calls back into to the web server to retrieve actual data to display within the user interface.

Orthogonal to single-data source web applications and their architecture, a new class of application has also emerged on the web, called mashups. Mashups are front-ends that aggregate and combine data that is available in a "pure data form" on the web – currently mostly are in the form of RSS/Atom feeds - and add value on top of the individual pieces of data.

This landscape makes it particularly interesting to talk about data services, the services that applications use to find and manipulate data on the web, regardless of the presentation technology used in the user interface or whether or not the front-end is hosted on the same site as the server hosting the data.

ADO.NET Data Services for the Web

The ADO.NET Data Services framework consists of a combination of patterns and libraries that enable the creation and consumption of data services for the web.

The goal of the ADO.NET Data Services framework is to facilitate the creation of flexible data services that are naturally integrated with the web. As such, ADO.NET Data Services use URIs to point to pieces of data and simple, well-known formats to represent that data, such as JSON and ATOM (XML-based feed format). This results in the data service being surfaced as a REST-style resource collection that is addressable with URIs and that agents can interact with using standard HTTP verbs such as GET, POST, PUT or DELETE.

In order for the system to understand and leverage semantics over the data that it is surfacing, ADO.NET Data Services models the data exposed through the data service using a model called the Entity Data Model (EDM), an Entity-Relationship derivative. This organizes the data in the form of instances of "entity types", or "entities", and the associations between them.

For relational data, ADO.NET Data Services supports exposing an EDM model created using the ADO.NET Entity Framework. For all other (ie. non relational) data sources or to use additional database access technologies (ex. LINQ to SQL) a mechanism is provided which enables any data source to be modeled as entities and associations (ie. described using an EDM schema) and exposed as a data service.

This article will discuss the key elements of the ADO.NET Data Services framework at a high level and the motivation behind its design.

>A simple addressing scheme for data with uniform URIs

To describe the system by example we will use a customer-tracking data service. The URI that will represent all of the Customer entities in the example ADO.NET Data Service would be:

http://myserver/data.svc/Customers

In this context, the "/Customers" part of the URI points to the Customers entity-set, which is the container for Customer instances. It is also possible to point to a single entity using a URI, for example:

http://myserver/data.svc/Customers(‘ALFKI’)

This URI results in a single Customer entity whose key, as defined in the EDM schema, has a value of "ALFKI".

If the Customer entity includes properties, those properties can be addressed individually. For example, the following URI represents the Contact Name of the Customer entity with key “ALFKI”:

http://myserver/data.svc/Customers(‘ALFKI’)/ContactName

Since EDM schemas not only describe the structure of the entity types, but also the associations between them, the system can leverage that information to provide a mechanism for association traversal. For example, if each of the customers in the data service has a set of Sales Orders associated with it, the following URI would represent the set of Sales Orders associated with the Customer whose key has a value of “ALFKI”.

http://myserver/data.svc/Customers(‘ALFKI’)/Orders

It may be the case that an application wishes to operate on an entities’ data as well as entities related to the target entity. For example, if an application wants to display a master-details style grid, it likely needs to efficiently retrieve both the target Customer and that Customers’ orders to display. The following URI addresses a single Customer and states the Customers’ orders and alternate contact information is to be returned inline in the same response from the data service.

http://myserver/data.svc/Customers(‘ALFKI’)?$expand=Orders,AlternateContacts

All of these URIs represent resources, such as "a customer" or "the sales orders associated with the customer with the key 'ALFKI'". In addition to key-based selection, the ADO.NET Data Service URIs allow for simple predicates to be included. This enables the representation of sets based on properties of the entities themselves, such as "the set of active sales orders for the customer with key 'ALFKI'", which in ADO.NET Data Service URI form is:

http://myserver/data.svc/Customers(‘ALFKI’)/Orders?$filter=Active eq true

The $filter operator used in the above example may also be used in conjunction with string, math or date functions such as in the following URI, which represents “the set of active orders made in the year 2007 for the customer ‘ALFKI’”:

http://myserver/data.svc/Customers(‘ALFKI’)/Orders?$filter=Active eq true and (year(OrderDate) eq 2007)

If the Sales Orders entities have other related entities, the URI can continue to drill down into the association graph, traversing associations and filtering each of the resulting sets.

ADO.NET Data Service URIs can also include control information in the query string, to adjust how data is presented to the caller. Common requirements for web applications include aspects such as being able to request data in pages and to sort the data by various properties. Continuing with the example, the following URI lists the same Sales Orders but ordered by the order date:

http://myserver/data.svc/Customers(‘ALFKI’)/Orders?$filter=Active eq true&$orderby=OrderDate

Paging over results is achieved by combining the parameters “skip” and “top”; for example, if listing 10 customers per page, the third page would be obtained with this URI:

http://myserver/data.svc/Customers?$skip=30&$top=10

For URIs that represent a specific entity instance or a specific association, not only is it possible to use HTTP GET to retrieve the entity, but the system also handles HTTP PUT to replace an entity (effectively updating the values in the data service), HTTP POST to create new entities and HTTP DELETE to eliminate entities from the data service and its underlying store. Additional actions are also supported and are detailed in the “Using ADO.NET Data Services” whitepaper.

>Format Independent

ADO.NET Data Services uses minimalistic formats to represent data, and supports more than one format to accommodate as many client agents as possible.

Currently ADO.NET Data Services can represent data in JSON (JavaScript Object Notation) and AtomPub (Atom Publishing Protocol) formats.

The default representation format is AtomPub and is generated using a fixed mapping of the entity structure to the XML elements in AtomPub with some added semantics to make use of the information provided by the EDM schema. For example, the customer URI from the example above was:

http://myserver/data.svc/Customers(‘ALFKI’)

If an HTTP GET request is sent to the URI above, the data service would respond with a payload similar to that shown below:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?> <entry xml:base="http://myserver/data.svc/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom"> <id>http://myserver/data.svc/Customers('ALFKI')</id> <updated>2008-08-31T21:36:21Z</updated> <title /> <author> <name /> </author> <category term="NorthwindModel.Customers" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <content type="application/xml"> <d:CustomerID>ALFKI</d:CustomerID> <d:CompanyName>Alfreds Futterkiste</d:CompanyName> <d:ContactName>Maria Anders</d:ContactName> <d:ContactTitle>Sales Representative</d:ContactTitle> <d:Address>Obere Str. 57</d:Address> <d:City>Berlin</d:City> <d:Region m:null="true" /> <d:PostalCode>12209</d:PostalCode> <d:Country>Germany</d:Country> <d:Phone>030-0074321</d:Phone> <d:Fax>030-0076545</d:Fax> </content> <link rel="edit" href="Customers('ALFKI')" title="Customers" /> <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Orders" title="Orders" href="Customers('ALFKI')/Orders" type="application/xml;type=feed" /> <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/CustomerDemographics" title="CustomerDemographics" href="Customers('ALFKI')/CustomerDemographics" type="application/xml;type=feed" /> </entry>

NOTE: The Customer entity in the example above contains both properties with primitive values such as "CompanyName" and properties that point to another resource such as "Orders".

The client agent can choose another format by simply changing the Accept header in the HTTP request (see HTTP RFC 2616 content type negotiation sections for details). For example, by setting the Accept header to "application/json" the client agent would obtain the following result in JSON format instead of the AtomPub format shown above:

{

“d”:{ __metadata: { uri: "Customers(\'ALFKI\')", type: "NorthwindModel.Customers" }, CustomerID: "ALFKI", CompanyName: "Alfreds Futterkiste", ContactName: "Maria Anders", ContactTitle: "Sales Representative", Address: "Obere Str. 57", City: "Berlin", Region: null, PostalCode: "12209", Country: "Germany", Phone: "030-0074321", Fax: "030-0076545", Orders: { __deferred: { uri: "Customers(\'ALFKI\')/Orders" } }

CustomerDemographics: { __deferred: { uri: "Customers(\'ALFKI\')/CustomerDemographics" } } }

}

While semantically equivalent, this version will integrate naturally and easily with JavaScript environments such as AJAX-based applications.

NOTE: The payload is contained within the fixed “d” property to mitigate unintended use of JSON payloads created by an ADO.NET Data Service.

>Uniform patterns enable reuse

The application of simple and clear semantics from the data model to URIs and payloads in various formats result in uniform patterns for interacting with data services.

These uniform patterns present the opportunity to create reusable components that build on top of the patterns and the semantics behind them and provide powerful abstractions for developers.

For example, an AJAX-enabled “grid” or “table” widget could take “/Customers” as its input, and would be able to list customer entities, show the list in pages, allow the user to filter based on simple criteria and to click on different columns to change the sort order. I would also be able to support in-place editing, creation and deletion of customer rows.

In addition to UI widgets, client-side libraries can be built to present higher-level programming interfaces to the data services. Again, the use of uniform patterns allows these libraries to surface very expressive constructs such as association traversal. The ADO.NET Data Services framework includes a client library for the .NET Framework and Silverlight that can expose a data service in terms of .NET objects and supports association traversal, identity resolution and other simple, yet powerful data programming constructs. In addition, the ADO.NET Data Services Framework includes a JavaScript library (available on www.codeplex.com) to simplify inserting, updating and deleting resources from AJAX applications.

>A good web citizen

ADO.NET Data Services are designed to leverage and integrate well with the web in various ways. The use of a REST-style interface with URIs to point to resources and content-types, negotiated through HTTP headers, are examples of this.

Going beyond that, ADO.NET Data Services has being designed to leverage other aspects such as the well-established HTTP caching infrastructure.

The ADO.NET Data Services framework also enables use of the regular authentication schemes supported by the execution environment that hosts a particular data service (typically Microsoft ASP.NET & Windows Communication Foundation).

>Storage Independence

ADO.NET Data Services are intended to expose data regardless of the underlying data source. In general, ADO.NET Data Services work by accepting an HTTP request for a resource identified by a URI, deserializing the request and then passing a representation of that request to an ADO.NET Data Service provider to execute the request on the underlying data store. This separation of the ADO.NET Data Services protocol (URI, payload format and interaction model) from the data store enables ADO.NET Data Services to expose data at the level of abstraction presented to it by the provider, which may be different from the schema of the underlying store. ADO.NET Data Services enables easily creating data services using:

Entity Framework data sources, which expose a conceptual model of data stored in relational databases (SQL, Oracle, DB2, MySql, etc)
Any + (interface to support update semantics for ADO.NET Data Services) implementation.This enables ADO.NET Data Services to expose any data store as REST-based endpoints regardless of its storage mechanism.

>Incorporating Business Logic

The sections above described how ADO.NET Data Services map a URI to each entity within an EDM-based conceptual model, which a developer defines and provides to a data service to expose as HTTP resources. This approach works well for classes of data (ex. blog articles) that can be served from the storage medium all the way to the application where it is likely to be directly presented to the end user. While such data exists, it is clear that not all data fits this mold. Some types of data for a variety of reasons (such as validation etc.) need to be accompanied by business logic that governs how the data is served.

ADO.NET Data Services enables a developer to go beyond simple REST semantics, where required, by defining service operations. Service operations enable a data service to define a method on the server, which just like all other ADO.NET Data Service resources, is identified by a URI. For example, the URI http://myserver/data.svc/MyFavoriteBooks?category=sports&$orderby=Title&$top=2&$skip=2 represents a call to the service operation named MyFavoriteBooks that takes a single category parameter. One of the interesting features of service operations is that the output of the service operation can be acted on using standard ADO.NET Data query string operators. This is shown in the query string of the example above where the $orderby, $top and $skip operators are applied to the results of the operation to sort and then page over the result set.

Another mechanism, known as Interceptors is also provided, which enables custom validation or access control logic to be plugged into the request/response processing pipeline of a data service. Interceptors allow an arbitrary method to be called when a particular CRUD action occurs on a given resource being exposed by ADO.NET Data Services. Such a method may then alter the data, restrict access based on the principle of the request or even terminate the operation.

Finally, a significant amount of access to relational databases occurs via stored procedures. ADO.NET Data Services supports the use of stored procedures in two ways:

ADO.NET Data Services supports the Entity Framework as a data source, which makes it simple to map entities exposed through a data service to stored procedures within a relational database
Service operations can be used to map stored procedures to entities exposed via an ADO.NET Data Service

>Metadata for REST Services

ADO.NET Data Services are simply resource stores with added semantics that reflect on the payloads and URIs. Like WSDL for SOAP-based web services, an ADO.NET Data Service is described using an Entity Data Model (EDM) conceptual schema. An EDM conceptual schema (http://msdn2.microsoft.com/en-us/library/bb399281(VS.90).aspx) is an XML document written using the conceptual schema definition language (CSDL) which describes entities and the associations among those entities. This description language was selected because its constructs enable rich expressivity for resource-based services and no equivalent language currently exists in the web community for REST services.

By virtue of having a well defined description mechanism for ADO.NET Data Services, opportunities in the tooling space exist where design-time tools can extract metadata from services and build front-ends, wrappers and metadata repositories used at runtime to facilitate consumptions of these services.

The following example CSDL document defines two entities: Customers and Orders, each with a few primitive type properties as well as a special “navigation property” which defines the association between the entities. In the example a 1-to-many association exists between Customer and Order entities. Valid URIs for a data service described using the CSDL below are:

All Customers: http://host/service.svc/Customers

All Orders, ordered by ship date: http://host/service.svc/Orders?$orderby=ShippedDate

Customer with key “ALFKI”: http://host/service.svc/Customers(‘ALFKI’)

Orders for the Customer with key “ALFKI”: http://host/service.svc/Customers(‘ALFKI’)/Orders

The metadata document for the service: http://host/service.svc/$metadata

<?xml version="1.0" encoding="utf-8" standalone="yes" ?> <edmx:Edmx Version="1.0" xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx"> <edmx:DataServices> <Schema Namespace="NorthwindModelApp" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://schemas.microsoft.com/ado/2006/04/edm"> <EntityContainer Name="NorthwindEntities" m:IsDefaultEntityContainer="true"> <EntitySet Name="Customers" EntityType="NorthwindModel.Customers" /> <EntitySet Name="Orders" EntityType="NorthwindModel.Orders" />

>Conclusion

Separation of presentation and data is an intrinsic characteristic of new web applications. This separation introduces the need and opportunity for data services for the web.

The ADO.NET Data Services framework is Microsoft’s technology for creating and consuming data services. These data services use the Entity Data Model (EDM) to model data in terms of entities. These entities are exposed as URI-addressable resources that can be accessed using standard HTTP requests. ADO.NET Data Services also establishes uniform patterns that UI widget frameworks and libraries can leverage to provide additional services on top of the data services.