Walkthrough: Creating an IQueryable LINQ Provider
Updated: July 2008
This advanced topic provides step-by-step instructions for creating a custom LINQ provider. When you are finished, you will be able to use the provider you create to write LINQ queries against the TerraServer-USA Web service.
The TerraServer-USA Web service provides an interface to a database of aerial images of the United States. It also exposes a method that returns information about locations in the United States, given part or all of a location name. This method, which is named GetPlaceList, is the method that your LINQ provider will call. The provider will use Windows Communication Foundation (WCF) to communicate with the Web service. For more information about the TerraServer-USA Web service, see Overview of the TerraServer-USA Web Services.
This provider is a relatively simple IQueryable provider. It expects specific information in the queries that it handles and it has a closed type system, exposing a single type to represent the result data. This provider examines only one type of method call expression in the expression tree that represents the query, that is the innermost call to Where. It extracts the data that it must have in order to query the Web service from this expression. It then calls the Web service and inserts the returned data into the expression tree in the place of the initial IQueryable data source. The rest of the query execution is handled by the Enumerable implementations of the standard query operators.
The code examples in this topic are provided in C# and Visual Basic.
This walkthrough illustrates the following tasks:
Creating the project in Visual Studio.
Implementing the interfaces that are required for an IQueryable LINQ provider: IQueryable<T>, IOrderedQueryable<T>, and IQueryProvider.
Adding a custom .NET type to represent the data from the Web service.
Creating a query context class and a class that obtains data from the Web service.
Creating an expression tree visitor subclass that finds the expression that represents the innermost call to the Queryable.Where method.
Creating an expression tree visitor subclass that extracts information from the LINQ query to use in the Web service request.
Creating an expression tree visitor subclass that modifies the expression tree that represents the complete LINQ query.
Using an evaluator class to partially evaluate an expression tree. This step is necessary because it translates all local variable references in the LINQ query into values.
Creating an expression tree helper class and a new exception class.
Testing the LINQ provider from a client application that contains a LINQ query.
Adding more complex query capabilities to the LINQ provider.
Note:The LINQ provider that this walkthrough creates is available as a sample. See LINQ to TerraServer Provider Sample for more information.
You need the following components to complete this walkthrough:
Visual Studio 2008
Note: |
|---|
| Your computer might show different names or locations for some of the Visual Studio user interface elements in the following instructions. The Visual Studio edition that you have and the settings that you use determine these elements. For more information, see Visual Studio Settings. |
To create the project in Visual Studio
In Visual Studio, create a new Class Library application. Name the project LinqToTerraServerProvider.
In Solution Explorer, select the Class1.cs (or Class1.vb) file and rename it to QueryableTerraServerData.cs (or QueryableTerraServerData.vb). In the dialog box that pops up, click Yes to rename all references to the code element.
You create the provider as a Class Library project in Visual Studio because executable client applications will add the provider assembly as a reference to their project.
To add a service reference to the Web service
In Solution Explorer, right-click the LinqToTerraServerProvider project and click Add Service Reference.
The Add Service Reference dialog box opens.
In the Address box, type http://terraserver.microsoft.com/TerraService2.asmx.
In the Namespace box, type TerraServerReference and then click OK.
The TerraServer-USA Web service is added as a service reference so that the application can communicate with the Web service by way of Windows Communication Foundation (WCF). By adding a service reference to the project, Visual Studio generates an app.config file that contains a proxy and an endpoint for the Web service. For more information, see Introduction to Windows Communication Foundation Services in Visual Studio.
You now have a project that has a file that is named app.config, a file that is named QueryableTerraServerData.cs (or QueryableTerraServerData.vb), and a service reference named TerraServerReference.
To create a LINQ provider, at a minimum you must implement the IQueryable<T> and IQueryProvider interfaces. IQueryable<T> and IQueryProvider are derived from the other required interfaces; therefore, by implementing these two interfaces, you are also implementing the other interfaces that are required for a LINQ provider.
If you want to support sorting query operators such as OrderBy and ThenBy, you must also implement the IOrderedQueryable<T> interface. Because IOrderedQueryable<T> derives from IQueryable<T>, you can implement both of these interfaces in one type, which is what this provider does.
To implement System.Linq.IQueryable`1 and System.Linq.IOrderedQueryable`1
In the file QueryableTerraServerData.cs (or QueryableTerraServerData.vb), add the following code.
The IOrderedQueryable<T> implementation by the QueryableTerraServerData class implements three properties declared in IQueryable and two enumeration methods declared in IEnumerable and IEnumerable<T>.
This class has two constructors. The first constructor is called from the client application to create the object to write the LINQ query against. The second constructor is called internal to the provider library by the code in the IQueryProvider implementation.
When the GetEnumerator method is called on an object of type QueryableTerraServerData, the query that it represents is executed and the results of the query are enumerated.
This code, except for the name of the class, is not specific to this TerraServer-USA Web service provider. Therefore, it can be reused for any LINQ provider.
To implement System.Linq.IQueryProvider
Add the TerraServerQueryProvider class to your project.
The query provider code in this class implements the four methods that are required to implement the IQueryProvider interface. The two CreateQuery methods create queries that are associated with the data source. The two Execute methods send such queries off to be executed.
The non-generic CreateQuery method uses reflection to obtain the element type of the sequence that the query it creates would return if it was executed. It then uses the Activator class to construct a new QueryableTerraServerData instance that is constructed with the element type as its generic type argument. The result of calling the non-generic CreateQuery method is the same as if the generic CreateQuery method had been called with the correct type argument.
Most of the query execution logic is handled in a different class that you will add later. This functionality is handled elsewhere because it is specific to the data source being queried, whereas the code in this class is generic to any LINQ provider. To use this code for a different provider, you might have to change the name of the class and the name of the query context type that is referenced in two of the methods.
You will need a .NET type to represent the data that is obtained from the Web service. This type will be used in the client LINQ query to define the results it wants. The following procedure creates such a type. This type, named Place, contains information about a single geographical location such as a city, a park, or a lake.
This code also contains an enumeration type, named PlaceType, that defines the various types of geographical location and is used in the Place class.
To create a custom result type
Add the Place class and the PlaceType enumeration to your project.
The constructor for the Place type simplifies creating a result object from the type that is returned by the Web service. While the provider can return the result type defined by the Web service API directly, it would require client applications to add a reference to the Web service. By creating a new type as part of the provider library, the client does not have to know about the types and methods that the Web service exposes.
This provider implementation assumes that the innermost call to Queryable.Where contains the location information to use to query the Web service. The innermost Queryable.Where call is the where clause (Where clause in Visual Basic) or Queryable.Where method call that occurs first in a LINQ query, or the one nearest to the "bottom" of the expression tree that represents the query.
To create a query context class
Add the TerraServerQueryContext class to your project.
This class organizes the work of executing a query. After finding the expression that represents the innermost Queryable.Where call, this code retrieves the lambda expression that represents the predicate that was passed to Queryable.Where. It then passes the predicate expression to a method to be partially evaluated, so that all references to local variables are translated into values. Then it calls a method to extract the requested locations from the predicate, and calls another method to obtain the result data from the Web service.
In the next step, this code copies the expression tree that represents the LINQ query and makes one modification to the expression tree. The code uses an expression tree visitor subclass to replace the data source that the innermost query operator call is applied to with the concrete list of Place objects that were obtained from the Web service.
Before the list of Place objects is inserted into the expression tree, its type is changed from IEnumerable to IQueryable by calling AsQueryable. This type change is necessary because when the expression tree is rewritten, the node that represents the method call to the innermost query operator method is reconstructed. The node is reconstructed because one of the arguments has changed (that is, the data source that it is applied to). The Call(Expression, MethodInfo, IEnumerable<Expression>) method, which is used to reconstruct the node, will throw an exception if any argument is not assignable to the corresponding parameter of the method that it will be passed to. In this case, the IEnumerable list of Place objects would not be assignable to the IQueryable parameter of Queryable.Where. Therefore, its type is changed to IQueryable.
By changing its type to IQueryable, the collection also obtains an IQueryProvider member, accessed by the Provider property, that can create or execute queries. The dynamic type of the IQueryable°Place collection is EnumerableQuery, which is a type that is internal to the System.Linq API. The query provider that is associated with this type executes queries by replacing Queryable standard query operator calls with the equivalent Enumerable operators, so that effectively the query becomes a LINQ to Objects query.
The final code in the TerraServerQueryContext class calls one of two methods on the IQueryable list of Place objects. It calls CreateQuery if the client query returns enumerable results, or Execute if the client query returns a non-enumerable result.
The code in this class is very specific to this TerraServer-USA provider. Therefore, it is encapsulated in the TerraServerQueryContext class instead of being inserted directly into the more generic IQueryProvider implementation.
The provider you are creating requires only the information in the Queryable.Where predicate to query the Web service. Therefore, it uses LINQ to Objects to do the work of executing the LINQ query by using the internal EnumerableQuery type. An alternative way to use LINQ to Objects to execute the query is to have the client wrap the part of the query to be executed by LINQ to Objects in a LINQ to Objects query. This is accomplished by calling AsEnumerable<TSource> on the rest of the query, which is the part of the query that the provider requires for its specific purposes. The advantage of this kind of implementation is that the division of work between the custom provider and LINQ to Objects is more transparent.
Note: |
|---|
The provider presented in this topic is a simple provider that has minimal query support of its own. Therefore, it relies heavily on LINQ to Objects to execute queries. A complex LINQ provider such as LINQ to SQL may support the whole query without handing any work off to LINQ to Objects. |
To create a class to obtain data from the Web service
Add the WebServiceHelper class (or module in Visual Basic) to your project.
This class contains the functionality that obtains data from the Web service. This code uses a type named TerraServiceSoapClient, which is auto-generated for the project by Windows Communication Foundation (WCF), to call the Web service method GetPlaceList. Then, each result is translated from the return type of the Web service method to the .NET type that the provider defines for the data.
This code contains two checks that enhance the usability of the provider library. The first check limits the maximum time that a client application will wait for a response by limiting the total number of calls that are made to the Web service, per query, to five. For each location that is specified in the client query, one Web service request is generated. Therefore, the provider throws an exception if the query contains more than five locations.
The second check determines whether the number of results returned by the Web service is equal to the maximum number of results that it can return. If the number of results is the maximum number, it is likely that the results from the Web service are truncated. Instead of returning an incomplete list to the client, the provider throws an exception.
To create the visitor that finds the innermost Where method call expression
Add the ExpressionVisitor class to your project. This code is available in How to: Implement an Expression Tree Visitor. Add using directives (Imports statements in Visual Basic) to the file for the following namespaces: System.Collections.Generic, System.Collections.ObjectModel and System.Linq.Expressions.
Add the InnermostWhereFinder class, which inherits the ExpressionVisitor class, to your project.
This class inherits the base expression tree visitor class to perform the functionality of finding a specific expression. The base expression tree visitor class is designed to be inherited and specialized for a specific task that involves traversing an expression tree. The derived class overrides the VisitMethodCall method to seek out the expression that represents the innermost call to Where in the expression tree that represents the client query. This innermost expression is the expression that the provider extracts the search locations from.
To create the visitor that extracts data to query the Web service
Add the LocationFinder class to your project.
This class is used to extract location information from the predicate that the client passes to Queryable.Where. It derives from the base expression tree visitor class and overrides only the VisitBinary method.
The expression tree visitor base class sends binary expressions, such as equality expressions like place.Name == "Seattle" (place.Name = "Seattle" in Visual Basic), to the VisitBinary method. In this overriding VisitBinary method, if the expression matches the equality expression pattern that can provide location information, that information is extracted and stored in a list of locations.
This class uses an expression tree visitor to find the location information in the expression tree because a visitor is designed for traversing and examining expression trees. The resulting code is neater and less error-prone than if it had been implemented without using the visitor.
At this stage of the walkthrough, your provider supports only limited ways of supplying location information in the query. Later in the topic, you will add functionality to enable more ways of supplying location information.
To create the visitor that modifies the expression tree
Add the ExpressionTreeModifier class to your project.
This class derives from the base expression tree visitor class and overrides the VisitConstant method. In this method, it replaces the object that the innermost standard query operator call is applied to with a concrete list of Place objects.
The CopyAndModify method calls the base class implementation of the Visit method. This CopyAndModify method is necessary because the Visit method, which is protected (Protected in Visual Basic), cannot be called directly from the query context class.
This expression tree modifier class uses the expression tree visitor because the visitor is designed to traverse, examine, and copy expression trees. By deriving from the base expression tree visitor class, this class requires minimal code to perform its function.
The predicate that is passed to the Queryable.Where method in the client query may contain sub-expressions that do not depend on the parameter of the lambda expression. These isolated sub-expressions can and should be evaluated immediately. They could be references to local variables or member variables that must be translated into values.
The next class exposes a method, PartialEval(Expression), that determines which, if any, of the sub-trees in the expression can be evaluated immediately. It then evaluates those expressions by creating a lambda expression, compiling it, and invoking the returned delegate. Finally, it replaces the sub-tree with a new node that represents a constant value. This is known as partial evaluation.
To add a class to perform partial evaluation of an expression tree
This section contains code for three helper classes for your provider.
To add the helper class that is used by the System.Linq.IQueryProvider implementation
Add the TypeSystem class (or module in Visual Basic) to your project.
The IQueryProvider implementation that you added previously uses this helper class.
TypeSystem.GetElementType uses reflection to obtain the generic type argument of an IEnumerable<T> (IEnumerable(Of T) in Visual Basic) collection. This method is called from the non-generic CreateQuery method in the query provider implementation to supply the element type of the query result collection.
This helper class is not specific to this TerraServer-USA Web service provider. Therefore, it can be reused for any LINQ provider.
To create an expression tree helper class
Add the ExpressionTreeHelpers class to your project.
This class contains methods that can be used to determine information about and extract data from specific types of expression trees. In this provider, these methods are used by the LocationFinder class to extract location information from the expression tree that represents the query.
To add an exception type for invalid queries
Add the InvalidQueryException class to your project.
This class defines an Exception type that your provider can throw when it does not understand the LINQ query from the client. By defining this invalid query exception type, the provider can throw a more specific exception than just Exception from various places in the code.
You have now added all the pieces that are required to compile your provider. Build the LinqToTerraServerProvider project and verify that there are no compile errors.
You can test your LINQ provider by creating a client application that contains a LINQ query against your data source.
To create a client application to test your provider
Add a new Console Application project to your solution and name it ClientApp.
In the new project, add a reference to the provider assembly.
Drag the app.config file from your provider project to the client project. (This file is necessary for communicating with the Web service.)
Note:In Visual Basic, you may have to click the Show All Files button to see the app.config file in Solution Explorer.
Add the following using statements (Imports statement in Visual Basic) to the Program.cs (or Module1.vb in Visual Basic) file:
using System; using System.Linq; using LinqToTerraServerProvider;
Imports LinqToTerraServerProviderIn the Main method in the file Program.cs (or Module1.vb in Visual Basic), insert the following code:
QueryableTerraServerData<Place> terraPlaces = new QueryableTerraServerData<Place>(); var query = from place in terraPlaces where place.Name == "Johannesburg" select place.PlaceType; foreach (PlaceType placeType in query) Console.WriteLine(placeType);
Dim terraPlaces As New QueryableTerraServerData(Of Place) Dim query = From place In terraPlaces _ Where place.Name = "Johannesburg" _ Select place.PlaceType For Each placeType In query Console.WriteLine(placeType.ToString()) Next
This code creates a new instance of the IQueryable<T> type that you defined in your provider, and then queries that object by using LINQ. The query specifies a location to obtain data on by using an equality expression. Because the data source implements IQueryable, the compiler translates the query expression syntax into calls to the standard query operators defined in Queryable. Internally, these standard query operator methods build an expression tree and call the Execute or CreateQuery methods that you implemented as part of your IQueryProvider implementation.
Build ClientApp.
Set this client application as the "StartUp" project for your solution. In Solution Explorer, right-click the ClientApp project and select Set as StartUp Project.
Run the program and view the results. There should be approximately three results.
The provider that you have to this point provides a very limited way for clients to specify location information in the LINQ query. Specifically, the provider is only able to obtain location information from equality expressions such as Place.Name == "Seattle" or Place.State == "Alaska" (Place.Name = "Seattle" or Place.State = "Alaska" in Visual Basic).
The next procedure shows you how to add support for an additional way of specifying location information. When you have added this code, your provider will be able to extract location information from method call expressions such as place.Name.StartsWith("Seat").
To add support for predicates that contain String.StartsWith
In the LinqToTerraServerProvider project, add the VisitMethodCall method to the LocationFinder class definition.
Recompile the LinqToTerraServerProvider project.
To test the new capability of your provider, open the file Program.cs (or Module1.vb in Visual Basic) in the ClientApp project. Replace the code in the Main method with the following code:
QueryableTerraServerData<Place> terraPlaces = new QueryableTerraServerData<Place>(); var query = from place in terraPlaces where place.Name.StartsWith("Lond") select new { place.Name, place.State }; foreach (var obj in query) Console.WriteLine(obj);
Dim terraPlaces As New QueryableTerraServerData(Of Place) Dim query = From place In terraPlaces _ Where place.Name.StartsWith("Lond") _ Select place.Name, place.State For Each obj In query Console.WriteLine(obj) Next
Run the program and view the results. There should be approximately 29 results.
The next procedure shows you how to add functionality to your provider to enable the client query to specify location information by using two additional methods, specifically Enumerable::Contains and List<T>::Contains. When you have added this code, your provider will be able to extract location information from method call expressions in the client query such as placeList.Contains(place.Name), where the placeList collection is a concrete list supplied by the client. The advantage of letting clients use the Contains method is that they can specify any number of locations just by adding them to placeList. Varying the number of locations does not change the syntax of the query.
To add support for queries that have the Contains method in their 'where' clause
In the LinqToTerraServerProvider project, in the LocationFinder class definition, replace the VisitMethodCall method with the following code:
This method adds each string in the collection that Contains is applied to, to the list of locations to query the Web service with. A method named Contains is defined in both Enumerable and List<T>. Therefore, the VisitMethodCall method must check for both of these declaring types. Enumerable.Contains is defined as an extension method; therefore the collection it is applied to is actually the first argument to the method. List.Contains is defined as an instance method; therefore the collection it is applied to is the receiving object of the method.
Recompile the LinqToTerraServerProvider project.
To test the new capability of your provider, open the file Program.cs (or Module1.vb in Visual Basic) in the ClientApp project. Replace the code in the Main method with the following code:
QueryableTerraServerData<Place> terraPlaces = new QueryableTerraServerData<Place>(); string[] places = { "Johannesburg", "Yachats", "Seattle" }; var query = from place in terraPlaces where places.Contains(place.Name) orderby place.State select new { place.Name, place.State }; foreach (var obj in query) Console.WriteLine(obj);
Dim terraPlaces As New QueryableTerraServerData(Of Place) Dim places = New String() {"Johannesburg", "Yachats", "Seattle"} Dim query = From place In terraPlaces _ Where places.Contains(place.Name) _ Order By place.State _ Select place.Name, place.State For Each obj In query Console.WriteLine(obj) Next
Run the program and view the results. There should be approximately 5 results.
This walkthrough topic showed you how to create a LINQ provider for one method of a Web service. If you want to pursue additional development of a LINQ provider, consider these possibilities:
Enable the LINQ provider to handle other ways of specifying a location in the client query.
Investigate the other methods that the TerraServer-USA Web service exposes, and create a LINQ provider that interfaces with one of those methods.
Find a different Web service that you are interested in, and create a LINQ provider for it.
Create a LINQ provider for a data source other than a Web service.