Service Compatibility

Microsoft Corporation

February 2008

Summary: This article describes what a tester should consider when updating services in a set of connected services. (6 printed pages)


Testing Goals


Test Requirements


More Information


In a continuously running Internet portal (for example,,, or, multiple services typically are involved. As new versions of services are released, they are introduced into production via a rollout, which is the replacement of the current version on a computer with a new version. Similarly, "rollback" changes to the previous version from the current version. Before we roll out a new version, it must undergo testing to verify that it is a safe release. Beyond all of the functionality, performance, security, stress, and other tests, we have to look at compatibility.

Service-compatibility testing verifies that:

  • Rollout of the service has no negative effects.
  • Use of the new service has no negative effects.
  • Service can be rolled back without negative effects.

Most services have multiple instances running in production. The load-balancing mechanisms might direct a call for service to any instance. Thus, during a rollout, two versions of the service must coexist—say, vN and vN+1. Callers to vN must be able to call vN +1 and get equivalent behavior. Similarly, vN +1 calls to other (older) services must demand no more than what vN did.

So, assuming a steady state system with no rollouts currently underway, we might have the following:


Note that client and server are just to indicate who initiates the call (caller) and who receives the call (callee). Furthermore, the client or server might be a partner that is not part of your feature team or part of your product. Note that the same is true when the communication is more indirect via shared data stores (for example, log writers/readers)—only even more complicated, as Y can "communicate with itself" via a data store from which it reads and writes (for example, persistent state). So, think also about your partner consumers and producers.


During rollout of service Y from vN to vN +1 (denoted as Y+), we have for the service-message protocols the following:


Typically, changing only Y means that there is no protocol change, because neither the caller (X) nor the callee (Z) of Y has changed. However, Y+ might rollout with a protocol change in anticipation of future or concurrent (as shown later) rollouts of X or Z.

Protocols can be updated in myriad ways. Two common update methods are transparent extension (compatible evolution) and explicit versioning (big bang). Transparent extension can occur by adding optional parameters. For example, say that X can send only three parameters to Y, and Y can return only two. Y+ must return only the same two parameters when dealing with X, but it can allow an optional fourth parameter (typically, coming only from X+). If it receives an optional fourth parameter, it can then return three parameters.

Note  Some protocols are built to "ignore" extra parameters—allowing Y+ to return excess parameters to X. However, with such "lazy" protocols, there are security implications.

Similarly, Y might be allowed to send only two parameters to Z, but Y+ might be built to accept an optional third parameter from Z (typically, a Z+1 version). Explicit versioning means that the collection of messages between two services is collected under a single version umbrella (for example, a versioned namespace). Changing any message means creation of a new version of the collection of messages. Typically, the start of a conversation between services identifies the version of the messages that are to be exchanged. It is possible also just to version every message.

For more coverage about this topic, see "5.2 Web Service Versioning" ( in the W3 working group WSDL 2.0 primer.

With data sharing, typically, Y+ might be rolled out able to "understand" new XY or ZY formats, but the only format to which it can change immediately might be the vN+1 format of its own data store.

How the versioning is done depends on the data-flow relationship. If Y merely is writing a log file, Y+ might intermix new (versioned) log records (assuming that the log reader is updated before or during the rollout!) into the same data store. If each instance of Y stores some independent state (for example, a cache), the Y would read/write to its store in vN format, while Y+ could read/write to its sore in vN+1 format. In rarer cases, the format is extensible, such that Y ignores extra or changed data in vN+1 formats. Y+ also might be rolled out with the vN+1 format disabled; then, after all instances are upgraded to Y+, a configuration change is used to allow vN+1 writing.


However, the preceding is just too easy. Frequently, along with your service Y, the surrounding services X and Z might be rolled out also as part of the release train. In this case, the situation looks something more like the following:


In that case, the explicit versioning model, X/Y, X/Y+, and X+/Y all would use version N of the protocol, while X+/Y+ could use a totally different N+1 version of the protocol. Similarly, Y/Z, Y+/Z, and Y/Z+ would use version M of their protocol, while Y+/Z+ could use a totally different M+1 version of their protocol.

Additionally, as alluded to earlier, it is possible for data formats to change as a release train rolls out. You must think about the producer and consumers of data in the same rollout.


A phased rollout should make sure that all data consumers are rolled out prior to data producers.

We conduct a service-compatibility test prior to each release, to ensure compatibility across versions. We also take this opportunity to ensure that, for each of the services, we are able to rollback to the prior version.

Test should be done on a final release candidate (RC). A final RC is a build that has no crashes (or dumps) or active bugs, and on in which resolved bugs are triaged for any risk/impact that is pertinent to the backward-compatibility.

Verify Action item


Establish a test environment that mirrors production.


Deploy a vN+1 release-candidate (RC) build to test the environment.

Can rollout the release train

Run tests that utilize the functionality of the services and their interactions during the rollout.

Rollout release-train services to half of the computers.


  • Each service must verify compatibility with both its callees and its callers.
  • Ensure that all functionality is intact on both upgraded and old environments.
  • Verify that the vN and vN +1 versions of service work with the vN and vN+1 data versions.
  • Verify that all vN+1 consumers of data are able to handle the vN and vN+1 of data.


Rollout the release train to the remaining computers.


Validate no new minidumps and that functionality tests are not showing errors.


Turn on any new functionality (upgrade) via configuration.


  • Each service must verify compatibility with its both its callees and its callers.
  • Create new data. (For example, generate new data, so that there is some in different states of an object life cycle during the upgrade.)
  • Is data rolled back also in case of a rollback?

Can rollback any specific service

Rollback each service individually.


  • Each service must verify compatibility with both its callees and its callers.

Can rollout any specific service

Rollout each service individually.


  • Each service must verify compatibility with both its callees and its callers.

Can rollback the release train

Rollback the release train.


  • Each service must verify compatibility with both its callees and its callers.

The italicized rows are only for services that use the additional step of turning on functionality via configuration, instead of just having it available in the new version (at rollout).

If new functionality becomes available just due to rollout, all interactions (protocols and data formats) that are due to old and new versions can be verified by rolling out just half when there are at least four computers total. Besides old and new interactions, we need two old versions to talk to each other and two new versions to talk to each other.

As the preceding table shows, tests for each service that verify that all of the callers and callees of the service behave normally must be provided and available. These tests will be run repeatedly for the various test bed states that were indicated earlier, so that the tests must be automated and relatively quick.

Make sure to test all new and old callee and caller scenarios.

It takes a lot of planning and testing to provide the confidence that updating services in a set of connected services can be done while providing site reliability. Testers must consider both direct message interactions and shared (for example, file) data or state information.

Community Additions