---

December 2009

Volume 24 Number 12

Generation Test - Automated Unit Tests for Legacy Code with Pex

By Nikhil Sachdeva | December 2009

In my previous life I was a consultant. One of my clients, a leading bank, wanted to automate its loan origination process. The bank already had a system in place that included a Windows-based application, a proprietary back end and a mainframe system that was also the heart of their solution. My job was to integrate the existing systems with a set of applications being developed for the accounts division.

The client had built a Web service that communicated with the mainframe. At first it seemed pretty simple. All I had to do was to hook into the service, get the information and pass it to the new accounts application. But it’s never that simple.

During implementation I found that the new systems expected a Loan Originator attribute, but the Web service GetLoanDetails method did not return that information. It turns out that the service was created years ago by a developer who’s no longer with the company. The bank has been using the service without any modifications because it has many layers and everyone is afraid of breaking something.

That Web service is legacy code.

Eventually, we built a lightweight service wrapper so the new systems could call it for any new information and continue using the old service as well. It would have been much easier to modify the existing service if a consistent and testable design had been followed.

Keeping Code Fresh

Legacy code is something that was developed in the past and is still used, but is hard to maintain and change. The reasons for sustaining these systems generally revolve around the cost and time involved in building a similar new system—though sometimes there is a lack of awareness about the future implications of current coding efforts.

The fact is that over a period of time code begins to rot. This can be because of requirement changes, a not-very-well-thought-through design, applied anti-patterns, or lack of appropriate tests. The end result is code that is hard to maintain and difficult to change.

There are many approaches to prevent your code from rotting, but one of the most effective can be to write code that is testable and then generate sufficient unit tests for the code. Unit tests act as agents that continuously probe the system for uncovered paths, identify troublesome errors, and provide indicators whether changes introduced into a subsystem have a good or bad effect on the overall software. Unit tests give confidence to the developer that the any code change will not introduce any regressions.

Having said that, creating and maintaining a good unit test suite can be a challenge in itself. It’s possible to end up writing more code for the test suite than the code under test. Another challenge is dependencies within the code. The more complex the solution, the more dependencies you are likely to find between classes. Mocks and stubs are a recognized way of removing these dependencies and testing the code in isolation, but these require additional knowledge and experience from the developer to create effective unit tests.

Pex to the Rescue

Pex (research.microsoft.com/projects/pex/) is a tool developed by Microsoft Research to automatically and systematically produce the minimal set of test inputs needed to execute a finite number of finite paths. Pex automatically produces a small test suite with high code and assertion coverage.

Pex finds interesting input-output values of your methods, which can then be saved as a small test suite with high code coverage. Pex performs a systematic analysis, hunting for boundary conditions, exceptions and assertion failures that you can debug right away. Pex also enables parameterized unit testing (PUT), an extension of unit testing that reduces test maintenance costs, and it utilizes dynamic symbolic execution to probe through the code under test to create a test suite covering most branches of execution.

A PUT is simply a method that takes parameters, calls the code under test and states assertions. A sample PUT looks like this:

void AddItem(List<int> list, int item) {
  list.Add(item);
  Assert.True(list[list.Count - 1] == item);
}

The PUT concept is derived from a broader terminology called data-driven testing (DDT), which has been used for a long time in traditional unit tests to make the tests repeatable. Because traditional unit tests are closed in nature, the only way to provide input values to them is through external sources such as an XML file, spreadsheets, or a database. While the DDT approach works well, there is additional overhead involved in maintaining and changing the external data file, and the inputs are dependent on the developer’s knowledge about the system.

Pex does not rely on external sources, and instead provides inputs to the test method by passing values to the corresponding PUT. Because the PUT is an open method, it can be arranged to accept any number of inputs. Moreover, Pex does not generate randomized values for the PUT. It relies on introspection of the method under test and generates meaningful values based on factors like boundary conditions, acceptable type values and a state-of-the-art constraint solver called Z3 (research.microsoft.com/en-us/um/redmond/projects/z3/). This ensures that all relevant paths of the method under test are covered.

The beauty of Pex is that it generates traditional unit tests from the PUT. These unit tests can be run directly in a unit testing framework like MSTest in Visual Studio without any modifications. Pex provides extensions to generate unit tests for frameworks like NUnit or xUnit.NET. You can also create your own custom extension. A Pex generated traditional unit test looks like this:

[TestMethod]
[PexGeneratedBy(typeof(TestClass))]
void AddItem01() {
  AddItem(new List<int>(), 0);
}

Dynamic symbolic execution is Pex’s answer to exploratory testing. Using this technique, Pex executes the code multiple times to understand the program behavior. It monitors the control and data flow and builds a constraint system for the test inputs.

Unit Testing with Pex

The first step is to create a PUT for the code under test. The PUT can be generated manually by the developer or by using the Visual Studio add-in for Pex. You can tailor the PUT by modifying parameters, adding Pex factories and stubs, integrating with mocks, adding assertions and so on. Pex factories and stubs are covered later in this article.

Currently the Visual Studio add-in for Pex creates PUTs only in C#, but the code under test can be in any .NET language.

The second step after the PUT has been set up is to run the Pex exploration. This is where Pex does its magic. It analyzes the PUT to identify what is being tested. It then starts the inspection of the code under test by passing through each branch and evaluating the possible input values. Pex repeatedly executes the code under test. After each run, it picks a branch that was not covered previously, builds a constraint system (a predicate over the test inputs) to reach that branch, then uses a constraint solver to determine new test inputs, if any. The test is executed again with the new inputs, and this process repeats.

On each run, Pex might discover new code and dig deeper into the implementation. In this way, Pex explores the behavior of the code.

While exploring the code, Pex produces a unit test suite that contains tests that cover all branches that Pex can exercise. These tests are standard 
unit test that can run in Visual Studio Test Editor. If you find a lower coverage for some sections, you might think of revising your code by refactoring and then applying the same cycle again to achieve higher code coverage and a more comprehensive test suite.

Pex can help in reducing the amount of effort and time involved when dealing with legacy code. Because Pex automatically explores the different branches and code paths, you don’t have to understand all details of the entire code base. Another benefit is that the developer works at the PUT level. Writing a PUT is often much simpler than writing closed unit tests because you focus on the problem scenario rather than on all possible test cases for the functionality.

Putting Pex to Work

Let’s use Pex on a piece of legacy code and see how it can help make the code more maintainable and easy to test.

Fabrikam, a leading manufacturer of basketballs and baseballs, provides an online portal where the user can view available products and place orders for those products. The inventory details are in a custom data store accessed through a Warehouse component that provides connectivity to the data store and operations like HasInventory and Remove. An Order component provides a Fill method that processes an order based on the product and quantity passed by the user.

The Order and WareHouse components are tightly coupled to each other. These components were developed years ago and no current employees have a thorough understanding of the system. No unit tests were created during development and, probably as a result, the components are very unstable. The current design is shown in Figure 1.

Figure 1 A Legacy Order Fulfillment System

The Fill method of the Order class looks like this:

public class Order {
  public bool Fill(Product product, int quantity) {
    // Check if WareHouse has any inventory
    Warehouse wareHouse = new Warehouse();
    if (wareHouse.HasInventory(product, quantity)) {
      // Subtract the quantity from the product in the warehouse
      wareHouse.Remove(product, quantity);
      return true;
    }
    return false;
  }
}

There are a few key items that require attention here. First, Order and Warehouse are tightly coupled. The classes depend on implementations that make them less extensible and difficult to use mock or stub frameworks. There are no unit tests available so any change might introduce regressions resulting in an unstable system.

The Warehouse component was written long ago and the current development team has no knowledge of how to change it or the implications of any changes. To make matters more complicated, Order cannot work with other implementations of Warehouse without modifications.

Let’s try to refactor the code and then use Pex to generate unit tests. I will refactor the Warehouse and Order objects, and then create unit tests for the Fill method of the Order class.

Refactoring legacy code is obviously a challenge. An approach in these cases might be to at least make the code testable so that sufficient unit tests can be generated. I will be applying only the bare minimal patterns that make the code testable.

The first problem here is that Order uses a specific implementation of Warehouse. This makes it difficult to decouple the Warehouse implementation from the Order. Let’s modify the code a bit to make it more flexible and testable.

I start by creating an interface IWareHouse and modify the Warehouse object to implement this interface. Any new Warehouse will require this interface to be implemented.

Because Order has a direct dependency on Warehouse, they are tightly coupled. I use dependency injection to open the class for extension of its behavior. Using this approach, an IWareHouse instance will be passed to Order at runtime. The new design is shown in Figure 2.

Figure 2 Revised System with IWareHouse

The new Order class is shown in Figure 3.

Figure 3 Revised Order Class

public class Order {
  readonly IWareHouse orderWareHouse;
  // Use constructor injection to provide a wareHouse object
  public Order(IWareHouse wareHouse) {
    this.orderWareHouse = wareHouse;
  }
  public bool Fill(Product product, int quantity) {
    // Check if WareHouse has any inventory
    if (this.orderWareHouse.HasInventory(product, quantity)) {
      // Update the quantity for the product
      this.orderWareHouse.Remove(product, quantity);
      return true;
    }
    return false;
  }
}

Creating Parameterized Unit Tests

Let’s now use Pex to generate tests for the refactored code. Pex provides a Visual Studio add-in that makes generating PUTs easy. Right-click the project, class, or method for which the PUTs need to be generated and click Pex | Create Parameterized Unit Test Stubs. I started by selecting the Fill method of the Order class.

Pex allows you to select an existing unit test project or to create a new one. It also gives you options to filter tests based on method or type name (see Figure 4).

Figure 4 Setting up a New Pex Project

Pex generates the following PUT for the Fill method.

[PexClass(typeof(Order))]
[TestClass]
public partial class OrderTest {
  [PexMethod]
  public bool Fill([PexAssumeUnderTest] Order target, 
    Product product, int quantity) {
    // Create product factory for Product
    bool result = target.Fill(product, quantity);
    return result;
  }
}

The OrderTest is not just a normal TestClass; it has been annotated with a PexClass attribute, indicating that it was created by Pex. There is currently no TestMethod as you would expect in a standard Visual Studio unit test. Instead, you have a PexMethod. This method is a parameterized unit test. Later, when you let Pex explore the code under test, it will create another partial class that contains the standard unit tests annotated with the TestMethod attributes. These generated tests will be accessible through the Visual Studio Test Editor.

Notice that the PUT for the Fill method takes three parameters.

[PexAssumeUnderTest] Order target

This is the class under test itself. The PexAssumeUnderTest attribute tells Pex that it should only pass non-null values of the exact specified type.

Product product

This is the Product base class. Pex will try to create instances of the product class automatically. For more granular control you can provide Pex with factory methods. Pex will use these factories to create instances of complex classes.

int quantity

Pex will provide values for the quantity based on the method under test. It will try to inject values that are meaningful for the test rather than junk values.

Pex Factories

As I mentioned before, Pex uses a constraint solver to determine new test inputs for parameters. The inputs can be standard .NET types or custom business entities. During exploration, Pex actually creates instances of these types so that the program under test can behave in different interesting ways. If the class is visible and has a visible default constructor, Pex can create an instance of the class. If all the fields are visible, it can generate values for them as well. However, if the fields are encapsulated with properties or not exposed to the outside world, Pex requires help to create the object so as to achieve a better code coverage.

Pex provides two hooks to create and link required objects to the Pex exploration. The user can provide factories for complex objects so that Pex can explore different object states. This is achieved through Pex Factory method. The types that can be created through such factories are called explorable types. We will be using this approach in this article.

The other approach is to define the invariants of the objects private fields so Pex can manufacture different object states directly.

Coming back to the example scenario, if you run a Pex exploration for the generated parameterized tests, the Pex Exploration Results window will show a message “2 Object Creations.” This is not an error. During exploration, Pex encountered a complex class (Order in this case) and created a default factory for that class. This factory was required by Pex to understand the program behavior better.

The default factory created by Pex is a vanilla implementation of the required class. You can tailor this factory to provide your own custom implementation. Click on Accept/Edit Factory to inject the code into your project (see Figure 5). Alternatively, you can create a static class with a static method that is annotated with the PexFactoryMethod attribute. While exploring, Pex will search in the test project for any static classes with methods with this attribute and use them accordingly.

Figure 5 Creating the Default Factory

OrderFactory looks like this:

public static partial class OrderFactory {
  [PexFactoryMethod(typeof(Order))]
  public static Order Create(IWareHouse wareHouseIWareHouse) {
        
    Order order = new Order(wareHouseIWareHouse);
    return order;
  }
}

If you write factory methods in other assemblies, you can tell Pex to use them in a declarative fashion by using the assembly level PexExplorableFromFactoriesFromType or PexExplorableFromFactoriesFromAssembly attributes, for example.

[assembly: PexExplorableFromFactoriesFromType(
  typeof(MyTypeInAnotherAssemblyContainingFactories))]

If Pex creates very few tests or fails to create interesting tests by just throwing a NullReferenceException on the object it should create, then this is a good indication that Pex might require a custom factory. Otherwise, Pex comes with a set of heuristics to create object factories that work in many cases.

Pex Stubs Framework

In software development, the notion of a test stub refers to a dummy implementation that can replace a possibly complex component to facilitate testing. While the idea of stubs is simple, most existing frameworks that can help to create and maintain dummy implementations are actually quite complex. The Pex team has developed a new lightweight framework, which they simply call Stubs. Stubs generates stub types for .NET interfaces and non-sealed classes.

In this framework, a stub of type T provides a default implementation of each abstract member of T, and a mechanism to dynamically specify a custom implementation of each member. (Optionally, stubs can also be generated for non-abstract virtual members.) The stub types are generated as C# code. The framework relies solely on delegates to dynamically specify the behavior of stub members. Stubs supports the .NET Framework 2.0 and higher and integrates with Visual Studio 2008 and higher.

In my example scenario the Order type has a dependency on the Warehouse object. Remember, I refactored the code to implement dependency injection so that Warehouse access can be provided from outside to the Order type. That comes in handy when creating the stubs.

Creating a stub is fairly simple. All you need is a .stubx file. If you created the test project through Pex, you should already have it. If not, this file can be created from within Visual Studio. Right-click on the test project and select Add New Item. A Stubs template is available (see Figure 6).

Figure 6 Creating a New Stub

The file appears as a standard XML file in Visual Studio. In the assembly element, specify the name of the assembly for which the stubs need to be created and save the .stubx file:

<Stubs xmlns="https://schemas.microsoft.com/stubs/2008/">
  <Assembly Name="FabrikamSports" />
</Stubs>

Pex will automatically create the necessary stub methods for all the types in the assembly.

The generated stub methods have corresponding delegate fields that provide hooks for stubbed implementations. By default, Pex will provide an implementation for the delegate. You can also provide a lambda expression to attach behavior to the delegate or use the PexChoose type to let Pex automatically generate values for the method.

For example, to provide choices for the HasInventory method, I can have something like this:

var wareHouse = new SIWareHouse() {
  HasInventoryProductInt32 = (p, q) => {
    Assert.IsNotNull(p);
    Assert.IsTrue(q > 0);
    return products.GetItem(p) >= q;
  }
};

In fact, using PexChoose is already the default behavior for stubs when using Pex, as the test project created by Pex contains the 
following assembly-level attribute:

[assembly: PexChooseAsStubFallbackBehavior]

The SIWareHouse type is generated by the Pex Stubs framework. It implements the IWareHouse interface. Let’s take a closer look at the code generated by Pex for the SIWareHouse stub. The source code for a .stubx file is created in a partial class with the name <StubsxFilename>.designer.cs as shown in Figure 7.

Figure 7 Pex-Generated IWareHouse Stub

/// <summary>Stub of method System.Boolean 
/// FabrikamSports.IWareHouse.HasInventory(
/// FabrikamSports.Product product, System.Int32 quantity)
/// </summary>
[System.Diagnostics.DebuggerHidden]
bool FabrikamSports.IWareHouse.HasInventory(
  FabrikamSports.Product product, int quantity) {
    
  StubDelegates.Func<FabrikamSports.Product, int, bool> sh
    = this.HasInventory;
  if (sh != (StubDelegates.Func<FabrikamSports.Product, 
    int, bool>)null)
    return sh.Invoke(product, quantity);
  else {
    var stub = base.FallbackBehavior;
    return stub.Result<FabrikamSports.Stubs.SIWareHouse, 
      bool>(this);
  }
}
/// <summary>Stub of method System.Boolean 
/// FabrikamSports.IWareHouse.HasInventory(
/// FabrikamSports.Product product, System.Int32 quantity)
/// </summary>
public StubDelegates.Func<FabrikamSports.Product, int, bool> HasInventory;

Stubs created a public delegate field for the HasInventory method and invoked it in the HasInventory implementation. If no implementation is available, Pex calls the FallBackBehaviour.Result method, which would use PexChoose if the [assembly: PexChooseAsStubFallbackBehavior] is present, and throws a StubNotImplementedException otherwise.

To use the stubbed implementation of IWareHouse, I will tweak the Parameterized unit test a bit. I already modified the Order class to be able to take an IWareHouse implementation in its constructor. I now create a SIWareHouse instance, then pass that to the Order class so it uses the custom implementations of the IWareHouse methods. The revised PUT is shown here:

[PexMethod]
public bool Fill(Product product, int quantity) {
  // Customize the default implementation of SIWareHouse
  var wareHouse = new SIWareHouse() {
    HasInventoryProductInt32 = (p, q) => 
    PexChoose.FromCall(this).ChooseValue<bool>(
    "return value")
  };
  var target = new Order(wareHouse);
  // act
  bool result = target.Fill(product, quantity);
  return result;
}

Stubs actually automatically provides default implementations for the stub methods, so you could have simply run the PUT without any modifications as well.

Parameterized Models

For a more granular control of the stubbed implementation, Pex supports a concept called a parameterized model. This is a way to write stubs that do not have one particular fixed behavior. The abstraction that Pex provides through this concept is that the developer does not need to worry about the variations of the implementation. Pex will explore different return values of the method based on how they are used by the code under test. Parameterized models are a powerful feature that allows you to take complete control over how the stubs should be processed while at the same time letting Pex evaluate the variant values for the input parameters.

A parameterized model for IWareHouse might look like the code in Figure 8.

Figure 8 Parameterized Model of IWareHouse

public sealed class PWareHouse : IWareHouse {
  PexChosenIndexedValue<Product, int> products;
  public PWareHouse() {
    this.products = 
      new PexChosenIndexedValue<Product, int>(
      this, "Products", quantity => quantity >= 0);
  }
  public bool HasInventory(Product product, int quantity) {
    int availableQuantity = this.products.GetItem(product);
    return quantity - availableQuantity > 0;
  }
  public void Remove(Product product, int quantity) {
    int availableQuantity = 
    this.products.GetItem(product); 
    this.products.SetItem(product, 
      availableQuantity - quantity);
  }
}

Essentially I have created my own stubbed implementation for IWareHouse, but notice that I do not provide values for Quantity and Product. Instead I let Pex generate those values. PexChosenIndexedValue automatically provides the values for the objects, allowing only one stubbed implementation with variant parameter values.

For simplicity I will let Pex provide the HasInventory implementation of the IWareHouse type. I will add code to the OrderFactory class that I created earlier. Every time an Order instance is created by Pex it will use a stubbed Warehouse instance.

Moles

So far I’ve focused on two principles—refactor your code to make it testable, then use Pex to generate unit tests. This approach lets you clean your code, eventually leading to more maintainable software. However, refactoring legacy code can be a big challenge in itself. There can be numerous organizational or technical constraints that might prevent a developer from refactoring the current source code. How do you deal with this?

In scenarios where legacy code is hard to refactor, the approach should be to at least create sufficient unit tests for the business logic so that you can verify the robustness of each module of the system. Mock frameworks like TypeMock (learn.typemock.com) have been around for a while. They let you create unit tests without actually modifying the code base. This approach proves very beneficial specifically for large legacy codebases.

Pex comes with a feature called Moles that lets you achieve the same goals. It allows you to generate Pex unit tests for legacy code without actually refactoring your source code. Moles are really meant to test otherwise untestable parts of the system, such as static methods and sealed classes.

Moles work in a similar fashion to stubs: Pex code-generates mole types that expose delegate properties for each method. You can attach a delegate, and then attach a mole. At that point, all your custom delegates get wired up magically by the Pex profiler.

Pex automatically creates moles for all static, sealed and public interfaces as specified in the .stubx file. A Pex Mole looks very similar to a Stubs type (see the code download for an example).

Using moles is fairly simple. You can provide implementations for the Mole methods and use them in your PUT. Notice that because Mole injects the stubbed implementations during runtime, you do not have to change your codebase at all to be able to generate unit tests using moles.

Let’s use moles on the legacy Fill method:

public class Order {
  public bool Fill(Product product, int quantity) {
    // Check if warehouse has any inventory
    Warehouse wareHouse = new Warehouse();
    if (wareHouse.HasInventory(product, quantity)) {
      // Subtract the quantity from the product 
      // in the warehouse
      wareHouse.Remove(product, quantity);
      return true;
    }
    return false;
  }
}

I create a PUT for the Fill method that leverages the Mole types (see Figure 9).

Figure 9 Using Mole Types on Fill

[PexMethod]
public bool Fill([PexAssumeUnderTest]Order target,
  Product product, int quantity) {
  var products = new PexChosenIndexedValue<Product, int>(
    this, "products");
  // Attach a mole of WareHouse type
  var wareHouse = new MWarehouse {
    HasInventoryProductInt32 = (p, q) => {
      Assert.IsNotNull(p);
      return products.GetItem(p) >= q;
    }
  };
  // Run the fill method for the lifetime of the mole 
  // so it uses MWareHouse
  bool result = target.Fill(product, quantity);
  return result;
}

MWareHouse is a Mole type that was created automatically by Pex when generating the stubs and moles via the .stubx file. I provide a custom implementation for the HasInventory delegate of the MWareHouse type, and then call the Fill method. Notice that nowhere do I provide an implementation of the Warehouse object to the Order type constructors. Pex will attach the MWareHouse instance to the Order type at run time. For the lifetime of the PUT any code written inside the block will leverage the MWareHouse type implementation wherever a Warehouse implementation will be required in the legacy code.

When Pex generates traditional unit tests that use moles, it attaches the attribute [HostType(“Pex”)] to them, so that they will execute with the Pex profiler, which will allow the moles to become active.

Putting It All Together

I talked about the various features of Pex and how to use them. Now it’s time to actually run the PUT and observe the results. To run an exploration for the Fill method of Order, simply right-click on the PUT and select Run Pex Exploration. You can optionally run the exploration on a class or the entire project.

While Pex exploration is running, a partial class is created along with the PUT class file. This partial class contains all the standard unit tests that Pex will generate for the PUT. For the Fill method, Pex generates standard unit tests using various test inputs. The tests are shown in Figure 10.

Figure 10 Pex-Generated Tests

[TestMethod]
[PexGeneratedBy(typeof(OrderTest))]
public void Fill15()
{
    Warehouse warehouse;
    Order order;
    Product product;
    bool b;
    warehouse = new Warehouse();
    order = OrderFactory.Create((IWareHouse)warehouse);
    product = new Product("Base ball", (string)null);
    b = this.Fill(order, product, 0);
    Assert.AreEqual<bool>(true, b);
}
[TestMethod]
[PexGeneratedBy(typeof(OrderTest))]
public void Fill16()
{
    Warehouse warehouse;
    Order order;
    Product product;
    bool b;
    warehouse = new Warehouse();
    order = OrderFactory.Create((IWareHouse)warehouse);
    product = new Product("Basket Ball", (string)null);
    b = this.Fill(order, product, 0);
    Assert.AreEqual<bool>(true, b);
}
[TestMethod]
[PexGeneratedBy(typeof(OrderTest))]
public void Fill17()
{
    Warehouse warehouse;
    Order order;
    Product product;
    bool b;
    warehouse = new Warehouse();
    order = OrderFactory.Create((IWareHouse)warehouse);
    product = new Product((string)null, (string)null);
    b = this.Fill(order, product, 1);
    Assert.AreEqual<bool>(false, b);
}

The key point to observe here is the variations for the Product type. Although I did not provide any factory for it, Pex was able to create different variations for the type.

Also note that the generated test contains an assertion. When a PUT returns values, Pex will embed the values that were returned at test generation time into the generated test code as assertions. As a result, the generated test is often able to detect breaking changes in the future, even if they don’t violate other assertions in the program code, or cause exceptions at the level of the execution engine.

The Pex exploration results window (see Figure 11) provides details of the unit tests generated by Pex. It also provides information about the factories Pex created and the events that occurred during the exploration. Notice in Figure 11 that two tests failed. Pex shows a NullReferenceException against both of them. This can be a common problem where you miss out placing validation checks in code paths that eventually might lead to exceptions when running in production.

Figure 11 Pex Exploration Results

Pex not only generates tests, but also analyzes the code for improvement. It provides a set of suggestions that can make the code even more stable. These suggestions are not just descriptive message, but actual code for the problem area. With a click of a button, Pex injects the code into the actual source file. Select the failed test in the Pex exploration result window. In the bottom-right corner, a button appears titled Add Precondition. Clicking this button adds the code into your source file.

These generated tests are normal MSTest unit tests and can be run from the Visual Studio Test Editor as well. If you open the editor, all the Pex generated tests will be available as standard unit tests. Pex can generate similar tests for other unit testing frameworks like NUnit and xUnit.

Pex also has built-in support for generating coverage reports. These reports provide comprehensive details around the dynamic coverage for the code under test. You can enable reports from Pex options in the Tools menu of Visual Studio, then open them by clicking Views | Report in the Pex menu bar after an exploration has finished.

Making Your Tests Future Ready

So far you’ve seen how Pex was able to generate code coverage for legacy code with minor refactoring of the source code. The beauty of Pex is that it relieves the developer from writing unit tests and generates them automatically, thereby reducing the overall testing effort.

One of the major pain points while unit testing is maintaining the test suite itself. As you progress in a project, you typically make lot of modifications to existing code. Because the unit tests are dependent on the source code, any code change impacts the corresponding unit tests. They might break or reduce the code coverage. Over a period of time keeping the test suite live becomes a challenge.

Pex comes in handy in this situation. Because Pex is based on an exploratory approach, it can search for any new changes in the code base and create new test cases based on them.

The prime purpose of regression testing is to detect whether modifications or additions to existing code have affected the codebase adversely, either through the introduction of functional bugs or creation of new error conditions.

Pex can generate a regression test suite automatically. When this regression test suite is executed in the future, it will detect breaking changes that cause assertions in the program code to fail, that cause exceptions at the level of the execution engine (NullReferenceException), or that cause assertions embedded in the generated tests to fail. Each time a Pex exploration is run fresh, unit tests are generated for the code under observation. Any change in behavior is picked up by Pex and the corresponding unit tests are generated for it.

Change Is Inevitable

Over a period of time, the developer team at Fabrikam realized that it made sense to have a ProductId attribute added to the Product class so that if the company adds new products to their catalog they can be uniquely identified.

Also the Order class was not saving the orders to a data store, so a new private method SaveOrders was added to the Order class. This method will be called by the Fill method when the product has some inventory.

The modified Fill method class looks like this:

public bool Fill(Product product, int quantity) {
  if (product == null) {
    throw new ArgumentException();
  }
  if (this.orderWareHouse.HasInventory(product, quantity)) {
    this.SaveOrder(product.ProductId, quantity);
    this.orderWareHouse.Remove(product, quantity);
  return true;
  }
  return false;
}

Because the signature of the Fill method was not changed, I do not need to revise the PUT. I simply run the Pex Exploration again. Pex runs the exploration, but this time it generates inputs using the new Product definition, utilizing the ProductId as well. It generates 
a fresh test suite, taking into account the changes made to the Fill method. The code coverage comes out to be 100 percent—ensuring that all new and existing code paths have been evaluated.

Additional unit tests are generated by Pex to test the variations of the added ProductId field and the changes made to the Fill method (see Figure 12). Here PexChooseStubBehavior sets the fallback behavior for the stubs; instead of just throwing a StubNotImplementedException, the stubbed method will call PexChoose to provide the possible return values. Running the tests in Visual Studio, the code coverage comes out to be 100 percent again.

Figure 12 Additional Pex-generated Unit Tests

[TestMethod]
[PexGeneratedBy(typeof(OrderTest))]
public void Fill12()
{
    using (PexChooseStubBehavior.NewTest())
    {
      SIWareHouse sIWareHouse;
      Order order;
      Product product;
      bool b;
      sIWareHouse = new SIWareHouse();
      order = OrderFactory.Create((IWareHouse)sIWareHouse);
      product = new Product((string)null, (string)null);
      b = this.Fill(order, product, 0);
      Assert.AreEqual<bool>(false, b);
    }
}
[TestMethod]
[PexGeneratedBy(typeof(OrderTest))]
public void Fill13()
{
    using (PexChooseStubBehavior.NewTest())
    {
      SIWareHouse sIWareHouse;
      Order order;
      Product product;
      bool b;
      sIWareHouse = new SIWareHouse();
      order = OrderFactory.Create((IWareHouse)sIWareHouse);
      product = new Product((string)null, (string)null);
      IPexChoiceRecorder choices = PexChoose.NewTest();
      choices.NextSegment(3)
          .OnCall(0, 
      "SIWareHouse.global::FabrikamSports.IWareHouse.HasInventory(Product, Int32)")
          .Returns((object)true);
      b = this.Fill(order, product, 0);
      Assert.AreEqual<bool>(true, b);
    }
}

Acknowledgements

I would like to thank Peli de Halleux and Nikolai Tillman for encouraging me to write this article, special gratitude to Peli for his tireless support, valuable comments and exhaustive review.

---

Nikhil Sachdeva is a software development engineer for the OCTO-SE team at Microsoft. You can contact him at blogs.msdn.com/erudition. You can also post your queries around Pex at social.msdn.microsoft.com/Forums/en/pex/threads.