April 2015

Volume 30 Number 4

Cutting Edge - Queryable Services

By Dino Esposito | April 2015

It’s getting more common for companies to expose their back-end business services as plain old HTTP endpoints. This type of architecture requires no information about databases and physical data models. Client applications don’t even need references to database-specific libraries such as Entity Framework. The physical location of the services is irrelevant and you can keep the back end on-premises or transparently move it to the cloud. 

As a solution architect or lead developer, there are two scenarios you should be prepared to face when adopting this strategy. The first is when you have no access whatsoever to the internal workings of the services. In that case, you aren’t even in a condition of asking for more or less data to fine-tune the performance of your client application.

The second scenario is when you’re also responsible for main­taining those back-end services and can influence the public API to a certain extent. In this article, I’ll focus primarily on the latter scenario. I’ll discuss the role of specific technologies to implement queryable services in a flexible way. The technology I’ll use is OData services on top of ASP.NET Web API. Nearly everything discussed in this article applies to existing ASP.NET platforms and ASP.NET 5 vNext, as well.

Sealed Back-End Services

Before getting into queryable service design, I’ll briefly explore that first scenario in which you have no control over available services. You’re given all the details you need to make calls to those services, but have no way to modify the amount and shape of the response.

Such sealed services are sealed for a reason. They’re part of the official IT back end of your company. Those services are part of the overall architecture and aren’t going to change lightheartedly. As more client applications depend on those services, chances are your company is considering versioning. Generally speaking, though, there has to be a compelling reason before any new release of those services is implemented.

If the sealed API is a problem for the client application you’re developing, the only thing you can do is wrap the original services in an additional proxy layer. Then you can use any tricks that serve your purposes, including caching, new data aggregates and inserting additional data. From an architectural standpoint, the resulting set of services then shifts from the infrastructure layer up to the domain services layer. It may even be higher at the application layer (see Figure 1).

Figure 1 From Sealed Services to More Flexible Application Services

The Read Side of an API

Modern Web applications are built around an internal API. In some cases, this API becomes public. It’s remarkable to consider that ASP.NET 5 vNext pushes an architecture in which ASP.NET MVC and the Razor engine provide the necessary infrastructure for generating HTML views.

ASP.NET Web API represents the ideal infrastructure for handling client requests coming from clients other than browsers and HTML pages. In other words, a new ASP.NET site is ideally devised as a thin layer of HTML around a possibly sealed set of back-end services. The team in charge of the Web application, though, is now also the owner of the back-end API, instead of the consumer. If anyone has an issue with that, you’ll hear his complaints or suggestions.

Most consumer API issues concern the quantity and quality of the data returned. The query side of an API is typically the trickiest to create because you never know which way your data is being requested and used in the long run. The command side of an API is usually a lot more stable because it depends on the business domain and services. Domain services do sometimes change, but at least they maintain a different and often slower pace.

Typically, you have a model behind an API. The query side of the API tends to reflect the model whether the API has a REST or an RPC flavor. Ultimately, the sore point of a read API is the format of the data it returns and the data aggregates it supports. This issue has a friendly name—data transfer objects (DTOs).

When you create an API service, you build it from an existing data model and expose it to the outside world through the same native or custom model. For years, software architects devised applications in a bottom-up manner. They always started from the bottom of a typically relational data model. This model traveled all the way up to the presentation layer.

Depending on the needs of the various client applications, some DTO classes were created along the way to ensure that presentation could deal with the right data in the right format. This aspect of software architecture and development is changing today under the effect of the growing role of client applications. While building the command side of a back-end API is still a relatively easy job, devising a single and general enough data model that suits all possible clients is a much harder job. The flexibility of the read API is a winning factor today because you never know which client applications your API will ever face.

Queryable Services

In the latest edition of the book, “Microsoft .NET: Architecting Applications for the Enterprise” (Microsoft Press, 2014), Andrea Saltarello and I formalized a concept we call layered expression trees (LET). The idea behind LET is application layer and domain services exchange IQueryable<T> objects whenever possible. This usually happens whenever the layers live in the same process space and don’t require serialization. By exchanging IQueryable<T>, you can defer any required query results from filter composition and data projection to the last minute. You can have these adapted on a per-application basis, instead of some hardcoded form of domain-level API.

The idea of LET goes hand-in-hand with the emerging CQRS pattern. This pushes the separation of the read stack from the command stack. Figure 2 illustrates the points of having an LET pattern and a bunch of queryable services in your architecture.

Figure 2 Queryable Objects Are Queried at the Last Minute

The primary benefit of LET is you don’t need any DTOs to carry data across layers. You still need to have view model classes at some point, but that’s a different story. You have to deal with view model classes as long as you have a UI to fill out with data. View model classes express the desired data layout your users expect. That’s the only set of DTO classes you’re going to have. Everything else from the level at which you physically query for data and up is served through IQueryable references.

Another benefit of LET and queryable services is the resulting queries are application-level queries. Their logic closely follows domain expert language. This makes it easier to map requirements to code and discuss alleged bugs or misunderstandings with customers. Most of the time, a quick look at the code helps you explain the logic. As an example, here’s what an LET query may look like:

var model = from i in db.Invoices
                        .ForBusinessUnit(buId)
                        .UnpaidInLast(30.Days)
  orderby i.PaymentDueDate
  select new UnpaidViewModel
    {
      ...
    };

From the database context of an Entity Framework root object, you query for all inbound invoices and select those relevant to a given business unit. Among those, you find those still unpaid a number of days past due terms.

The nice thing is IQueryable references are not real data. The query executes against the data source only when you exchange IQueryable for some IList. Filters you add along the way are simply WHERE clauses added to the actual query being run at some point. As long as you’re in the same process space, the amount of data transferred and held in memory is the bare minimum.

How does this affect scalability? The emerging trend for which the vNext platform has been optimized is you keep your Web back end as compact as possible. It will ideally be a single tier. You achieve scalability by replicating the unique tier through a variety of Microsoft Azure Web Roles. Having a single tier for the Web back end lets you use IQueryable everywhere and save yourself a bunch of DTO classes.

Implement Queryable Services

In the previous code snippet, I assumed your services are implemented as a layer around some Entity Framework database context. That’s just an example, though. You can also fully encapsulate the actual data provider under an ASP.NET Web API façade. That way, you have the benefit of an API that expresses the domain services capabilities and can still reach over HTTP, thus decoupling clients from a specific platform and technology.

Then you can create a Web API class library and host it in some ASP.NET MVC site, Windows service or even some custom host application. In the Web API project, you create controller classes that derive from ApiController and expose methods that return IQueryable<T>. Finally, you decorate each IQueryable method with the EnableQuery attribute. The now obsolete Queryable attribute works, as well. The key factor here is the EnableQuery attribute lets you append OData queries to the requested URL—something like this:

[EnableQuery]
public IQueryable<Customer> Get()
{
  return (from c in db.Customers select c);
}

This basic piece of code represents the beating heart of your API. It returns no data by itself. It lets clients shape up any return data they want. Look at the code in Figure 3 and consider it to be code in a client application.

Figure 3 Client Apps Can Shape the Data Returned

var api = "api/customers?$select=LastName";
var request = new HttpRequestMessage()
{
  RequestUri = new Uri(api)),
    Method = HttpMethod.Get,
};
var client = new HttpClient();
var response = client.SendAsync(request).Result;
if (response.IsSuccessStatusCode)
{
  var list = await
    response.Content.ReadAsAsync<IEnumerable<Customer>>();
  // Build view model object here
}

The $select convention in the URL determines the data projection the client receives. You can use the power of OData query syntax to shape up the query. For more on this, see bit.ly/15PVBXv.

For example, one client may only request a small subset of columns. Another client or a different screen in the same client may query a larger chunk of data. All this can happen without touching the API and creating tons of DTOs along the way. All you need is an OData queryable Web API service and the final view model classes to be consumed. Transferred data is kept to a minimum as only filtered fields are returned.

There are a couple of remarkable aspects to this. First, OData is a verbose protocol and couldn’t otherwise be given the role it plays. This means when you apply a $select projection, the JSON payload will still list all fields in the original IQueryable<T> (all fields of the original Customer class in the Get method). However, only the specified fields will hold a value.

Another point to consider is case sensitivity. This affects the $filter query element you use to add a WHERE clause to the query. You might want to call tolower or toupper OData functions (if supported by the OData client library you’re using) to normalize comparisons between strings.

Wrapping Up

Quite frankly, I never felt OData was worthy of serious consideration until I found myself—as the owner of a back-end API—in the middle of a storm of requests for different DTOs being returned from the same data model. Every request seemed legitimate and placed for a very noble cause—performance improvement.

At some point, it seemed all those clients wanted to do was “query” the back end in much the same way they would query a regular database table. Then I updated the back-end service to expose OData endpoints, thus giving each client the flexibility to download only the fields in which they were interested.

The type T of each IQueryable method is the key. It may or may not be the same type T you have in the physical model. It can match a plain database table or result from data aggregations done on the server-side and transparent to clients. When you apply OData, though, you let clients query on datasets of a known single entity, T, so why not give it a try?

Dino Esposito is the coauthor of “Microsoft .NET: Architecting Applications for the Enterprise” (Microsoft Press, 2014) and “Programming ASP.NET MVC 5” (Microsoft Press, 2014). A technical evangelist for the Microsoft .NET Framework and Android platforms at JetBrains and a frequent speaker at industry events worldwide, Esposito shares his vision of software at software2cents.wordpress.com and on Twitter at twitter.com/despos.

Thanks to the following technical expert for reviewing this article: Jon Arne Saeteras