SALES: 1-800-867-1380

Guidance on Efficiently Testing Azure Solutions

Updated: June 17, 2014

Author: Suren Machiraju

Reviewers: Jaime Alva Bravo and Steve Wilkins

After designing, coding, and deploying your Microsoft Azure solution, you sometimes find that it does not work. This article provides guidance on how to test your Microsoft Azure applications throughout the software development lifecycle. The testing scope includes the business logic and the comprehensive end-to-end scenario tests. This article demonstrates how to:

  • Develop unit tests for the business logic components while eliminating dependencies on Microsoft Azure components.

  • Design integration end-to-end tests.

  • Eliminate overhead for setup, initialization, cleanup, and teardown of the Microsoft Azure Service resources (for example, Queues) for every test run.

  • Eliminate the creation of duplicate infrastructures for ACS namespaces, Service Bus Queues, and other resources.

Additionally, this article provides an overview of the various testing technologies and techniques for testing your Microsoft Azure applications.

Updates in This Article

The following changes are new in this article:

  1. We use Visual Studio 2013.

  2. Microsoft Fakes in Visual Studio 2013 replaces Moles. We describe this feature in the “Isolating Code Under Test with Microsoft Fakes” article.

  3. In Visual Studio 2013, a new version of Code Coverage is now available and is invoked directly from Test Explorer. For a description, read the “Using Code Coverage to Determine How Much Code is being Tested” article.

  4. The Pex team has now released a lightweight version of Pex called Code Digger. For a description, see the “Microsoft Code Digger” article.

  5. Beginning with Visual Studio 2012, we no longer recommend the testing of private methods. For an explanation, read “My Take on Unit Testing Private Methods”.

The Types of Tests

There are two kinds of tests:

  • Unit Tests are narrowly focused tests that exercise a single, specific function. We refer to these tests as the Code Under Test or CUT. You must remove any dependencies that the CUT requires.

  • Integration Tests are broader tests that exercise multiple bits of functionality simultaneously. In many cases, these resemble unit tests, but they cover multiple feature areas and include multiple dependencies.

Overall, these tests focus on creating and using Test Doubles. We use the following types of Test Doubles:

  • Fakes are simulated objects that implement the same interface as the object that they represent. Fakes return predefined responses. A Fake contains a set of method stubs and serves as a replacement for what you build programmatically.

  • Stubs simulate the behavior of software objects.

  • Shims allow you to isolate your code from assemblies that are not part of your solution. They also isolate components of your solution from each other.

Once executed, these tests can verify state and behavior. For example, state includes the result after calling a method and returning a specific value. An example of behavior is calling a method in a certain order or a certain number of times.

The Azure Dependencies

One of the major goals of unit testing is to eliminate dependencies. For the Azure framework, these dependencies include the following:

  • Service Bus Queues.

  • Access Control Service.

  • Cache.

  • Azure Tables, Blobs, and Queues.

  • Azure SQL Database.

  • Azure Drive (formerly Cloud Drive).

  • Other Web Services.

When building tests for Azure applications, we replace these dependencies to focus the tests on exercising the logic.

The Service Bus Queues examples (including tools and techniques) that we discuss in this article also apply to all other dependencies.

The Testing Framework

To implement the testing framework for your Microsoft Azure applications, you need:

  • A unit testing framework to define and run your tests.

  • A mocking framework to help you isolate dependencies and build narrowly scoped unit tests.

  • Tools to help with automatic unit test generation for increased code coverage.

  • Other frameworks that can help with testable designs by taking advantage of dependency injection and applying the Inversion of Control Pattern (IoC).

Unit Testing with MS Test

Visual Studio includes a command-line utility, called MS Test, to execute unit tests created in Visual Studio. Visual Studio also includes a suite of project and item templates to support testing. You typically create a test project, and then add classes (known as test fixtures) adorned with the [TestClass] attribute. The classes contain methods adorned with the [TestMethod] attribute. Within MS Test, various windows within Visual Studio allow you to execute unit tests defined in the project. You can also review the results after you execute them.

noteNote
Visual Studio 2013 Express, Professional. and Test Professional editions do not contain MS Test.

In MS Test, unit tests follow an AAA pattern - Arrange, Act, and Assert.

  • Arrange - build any prerequisite objects, configurations, and all the other necessary preconditions and inputs the CUT needs.

  • Act - perform the actual, narrowly scoped test on the code.

  • Assert - verify that the expected results have occurred.

The MS Test framework libraries include the PrivateObject and PrivateType helper classes. These classes use reflection to make it easy to invoke nonpublic instance members or static members from within the unit test code.

Premium and Ultimate editions of Visual Studio include enhanced unit test tools. These tools integrate with MS Test. The tools also allow you to analyze the amount of code your unit tests exercise. Additionally, the tools color code the source code to indicate coverage. The feature is called Code Coverage.

Isolating Code with Microsoft Fakes

A goal of unit testing is to test in isolation. However, often you cannot test the code under test in isolation. Sometimes the code is not written for testability. To rewrite the code would be difficult because it relies on other libraries that are not easily isolated. For example, code that interacts with external environments is not easily isolated. A mocking framework helps you isolate both types of dependencies.

A list of mocking frameworks you might consider is available in the links section at the end of this article. This article focuses on how to use Microsoft Fakes.

Microsoft Fakes help you isolate the code you are testing by replacing other parts of the application with stubs or shims. These are small pieces of code that are under the control of your tests. By isolating your code for testing, you know that, if the test fails, the cause is there and not somewhere else. Stubs and shims also let you test your code even if other parts of your application are not working yet.

Fakes come in two forms:

  • A stub replaces a class with a small substitute that implements the same interface. To use stubs, you have to design your application so that each component depends only on interfaces and not on other components. By "component", we mean a class or group of classes that are designed and updated together and typically contained in an assembly.

  • A shim modifies the compiled code of your application at run time. Instead of making a specified method call, your application runs the shim code that your test provides. You can use shims to replace calls to assemblies that you cannot modify such as .NET assemblies.

Automated Test Creation with Code Digger

Code Digger analyzes possible execution paths through your .NET code. The result is a table in which each row shows a unique behavior of your code. The table helps you understand the behavior of the code, and it may also uncover hidden bugs.

To analyze your code in the Visual Studio editor, use the new context menu item Generate Inputs/Outputs Table to invoke Code Digger. Code Digger computes and displays input-output pairs. Code Digger systematically hunts for bugs, exceptions, and assertion failures.

Be default, Code Digger only works on public .NET code that resides in Portable Class Libraries. Later in this article, we will discuss how to configure Code Digger to explore other .NET projects.

Code Digger uses the Pex engine and Microsoft Research’s Z3 constraint solver to systematically analyze all branches in the code. Code Digger attempts to generate a test suite that achieves high code coverage.

Frameworks for Dependency Injection and Inversion of Control Containers

You can use Microsoft Unity for extensible dependency injection (DI) and inversion of control (IoC) containers. It supports interception, constructor injection, property injection, and method call injection. Microsoft Unity and similar tools help you build testable designs that let you insert your dependencies at all levels of your application. (We assume that your application was designed and built with dependency injection and one of these frameworks in mind.)

These frameworks are great for writing testable code and, ultimately, good code. However, they can be demanding in their upfront design requirements. We do not discuss DI and IoC containers in this article.

Testing Azure Solutions

In this section, we describe a solution that includes a website hosted on a Web Role. The website pushes messages to a Queue. Then a Worker Role processes messages from the Queue. We are interested in testing aspects of all three.

Unit Testing Controller Actions in a Web Role

Imagine you have a website that creates orders, and a Service Bus Queue creates a queue for processing these orders. The webpage resembles Figure 1:

Figure 1

Figure 1

When the user clicks Create, the Service Bus Queue posts the new order to the Create action on the associate controller. The action is implemented as follows:

noteNote
The MicrosoftAzureQueue class is a wrapper class that uses the Service Bus .NET APIs (such as MessageSender and MessageReceiver) to interact with Service Bus Queues.

private IMicrosoftAzureQueue queue;
public OrderController()
{
    queue = new MicrosoftAzureQueue();
}
public OrderController(IMicrosoftAzureQueue queue)
{
    this.queue = queue;
}
[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult Create([Bind(Include = "OrderId,Description")] Order order)
{
    try
    {
        if (ModelState.IsValid)
        {
            string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
            string queueName = "ProcessingQueue";
            queue.InitializeFromConnectionString(connectionString, queueName);
            queue.Send(order);
        }
        return View("OrderCreated");
    }
    catch (Exception ex)
    {
        Trace.TraceError(ex.Message);
        return View("Error");
    }            
}

Notice that the code retrieves configuration settings from CloudConfigurationManager and sends a message that contains the order to the queue. Also note that the Create action uses the following methods:

  • InitializeFromConnectionString (ConnectionString string, QueueName string)

  • Send (MicrosoftAzureQueue class)

We want to build detours for these methods by using Fakes to control their behavior and remove the dependencies on the real environment. Using Fakes eliminates the need to run the tests in the Azure emulator or to call the Service Bus Queue. We exercise the Create action on the controller to verify that the Order input is the one sent to the queue. The Send method checks that the input has the Order Id and Description as input to the action. It then checks that the OrderCreated view appears as a result.

Building a unit test for the implementation above using Fakes is easy. Within the Test project, right-click the assembly that contains the types you want to mock. Then select Add Fakes Assembly.

In the example, we select Microsoft.ServiceBus and select Add Fakes Assembly. An XML file named “Microsoft.ServiceBus.Fakes” is added to the test project. Repeat the action for the Microsoft.WindowsAzure.Configuration assembly.

When you build the Test project, references are added to the auto-generated mocked versions of these assemblies. In the example, the generated assemblies are “Microsoft.ServiceBus.Fakes” and “Microsoft.WindowsAzure.Configuration”.

Create a unit test method and apply the [TestCategory("With fakes")] attribute. Within the unit test, you use shims to isolate parts of your application from each other.

Using Shims to Isolate Your Application from Other Assemblies for Unit Testing

Shims are one of two technologies that the Microsoft Fakes Framework provides that let you easily isolate components under test from the environment. Shims divert calls to specific methods to code that you write as part of your test. Many methods return different results depending on external conditions. However, a shim is under the control of your test and can return consistent results at every call. This makes your tests much easier to write. Use shims to isolate your code from assemblies that are not part of your solution. To isolate components of your solution from each other, we recommend that you use stubs.

Using Stubs to Isolate Parts of Your Application from Each Other for Unit Testing

Stubs are one of two technologies that the Microsoft Fakes Framework provides that let you easily isolate a component you are testing from other components that it calls. A stub is a small piece of code that takes the place of another component during testing. The benefit of using a stub is that it returns consistent results, making the test easier to write. Additionally, you can run tests even if the other components are not working yet.

In our test case, we will use shims for CloudConfigurationManager and BrokeredMessage from Azure assemblies. We will use stubs for MicrosoftAzureQueue, which is a class in our solution.

[TestMethod]
[TestCategory("With fakes")]
public void Test_Home_CreateOrder()
{
    // Shims can be used only in a ShimsContext
    using (ShimsContext.Create())
    {
        // Arrange
        // Use shim for CloudConfigurationManager.GetSetting
        Microsoft.WindowsAzure.Fakes.ShimCloudConfigurationManager.GetSettingString = (key) =>
        {
            return "mockedSettingValue";
        };
                
        // Create the fake queue:
        // In the completed application, queue would be a real one:
        bool wasCreateFromConnString = false;
        Order orderSent = null;
        IMicrosoftAzureQueue queue =
                new OrderWebRole.Queue.Fakes.StubIMicrosoftAzureQueue() // Generated by Fakes.
                {
                    // Define each method:
                    // Name is original name + parameter types:
                    InitializeFromConnectionStringStringString = (connectionString, queueName) => {
                        wasCreateFromConnString = true;
                    },
                    SendOrder = (order) => {
                    orderSent = order;
                    }
                };

        // Component under test
        OrderController controller = new OrderController(queue);

        // Act
        Order inputOrder = new Order()
        {
            OrderId = System.Guid.NewGuid(),
            Description = "A mock order"
        };
        ViewResult result = controller.Create(inputOrder) as ViewResult;

        //Assert
        Assert.IsTrue(wasCreateFromConnString);
        Assert.AreEqual("OrderCreated", result.ViewName);
        Assert.IsNotNull(orderSent);
        Assert.AreEqual(inputOrder.OrderId, orderSent.OrderId);
        Assert.AreEqual(inputOrder.Description, orderSent.Description);
    }
}

Unit Testing Worker Roles that use Azure Service Bus Queues

The Web Role takes care of adding an order to the queue. Now, consider how to test a Worker Role that would process the orders by retrieving them from the Service Bus Queue. The Run method in our Worker Role periodically polls the queue for orders and processes them.

private IMicrosoftAzureQueue queue;
public WorkerRole()
{
    queue = new MicrosoftAzureQueue();
}

public WorkerRole(IMicrosoftAzureQueue queue)
{
    this.queue = queue;
}
public override void Run()
{
    try
    {
        string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
        string queueName = "ProcessingQueue";
               
        queue.InitializeFromConnectionString(connectionString, queueName);
            
        queue.CreateQueueIfNotExists();
              
        while (true)
        {
            Thread.Sleep(2000);
            //Retrieve order from Service Bus Queue  
            TryProcessOrder(queue);
        }
    }
    catch (Exception ex)
    {
        if (queue != null)
            queue.Close();
        System.Diagnostics.Trace.TraceError(ex.Message);
    }
}

We want a test to verify that the routine correctly retrieves a message. The following is the complete unit test for the Run method of the Worker Role.

[TestMethod]
public void Test_WorkerRole_Run()
{
    // Shims can be used only in a ShimsContext:
    using (ShimsContext.Create())
    {
        Microsoft.WindowsAzure.Fakes.ShimCloudConfigurationManager.GetSettingString = (key) =>
        {
            return "mockedSettingValue";
        };

        // Arrange 
        bool wasEnsureQueueExistsCalled = false;
        int numCallsToEnsureQueueExists = 0;

        // Create the fake queue:
        // In the completed application, queue would be a real one:
        bool wasConnectionClosedCalled = false;
        bool wasCreateFromConnString = false;
        bool wasReceiveCalled = false;
        int numCallsToReceive = 0;

        bool wasCompleteCalled = false;
        int numCallsToComplete = 0;
        IMicrosoftAzureQueue queue =
                new OrderWebRole.Queue.Fakes.StubIMicrosoftAzureQueue() // Generated by Fakes.
                {

                    // Define each method:
                    // Name is original name + parameter types:
                    InitializeFromConnectionStringStringString = (connectionString, queueName) =>
                    {
                        wasCreateFromConnString = true;
                    },
                    CreateQueueIfNotExists = () =>
                    {
                        wasEnsureQueueExistsCalled = true;
                        numCallsToEnsureQueueExists++;
                    },
                    Receive = () =>
                {
                    wasReceiveCalled = true;
                    if (numCallsToReceive >= 3) throw new Exception("Aborting Run");
                    numCallsToReceive++;
                    Order inputOrder = new Order()
                    {
                        OrderId = System.Guid.NewGuid(),
                        Description = "A mock order"
                    };
                    return new BrokeredMessage(inputOrder);
                },
                    Close = () =>
                    {
                        wasConnectionClosedCalled = true;
                    }

                };


        Microsoft.ServiceBus.Messaging.Fakes.ShimBrokeredMessage.AllInstances.Complete = (message) =>
        {
            wasCompleteCalled = true;
            numCallsToComplete++;
        };

        WorkerRole workerRole = new WorkerRole(queue);

        //Act
        workerRole.Run();

        //Assert
        Assert.IsTrue(wasCreateFromConnString);
        Assert.IsTrue(wasConnectionClosedCalled);
        Assert.IsTrue(wasEnsureQueueExistsCalled);
        Assert.IsTrue(wasReceiveCalled);
        Assert.AreEqual(1, numCallsToEnsureQueueExists);
        Assert.IsTrue(numCallsToReceive > 0);
        Assert.IsTrue(wasCompleteCalled);
        Assert.IsTrue(numCallsToComplete > 0);
        Assert.AreEqual(numCallsToReceive, numCallsToComplete);

    }
}

You should set the delegate of the AllInstances property of the generated Fake type. By using the delegate, any instance you create of the actual Fake type detours through any of the methods for which you have defined delegates.

In the example, we want to use the original instance’s Run method, but provide detours for the CreateQueue and TryProcessOrder instance methods. In the code, we throw an exception so that we exit the infinite loop that the Run method maintains at a predetermined time.

You may ask, why not just use the MessageSender/MessageReceiver and related classes from the Service Bus SDK directly instead of inserting a helper type? To completely isolate the code so that it does not call the real-world Service Bus, there are two choices:

  • Write Fakes that inherit from the abstract classes in the Microsoft.ServiceBus namespace.

  • Let Fakes create mock types for them all.

The problem with either approach is complexity. With both approaches, you ultimately find yourself reflecting into classes like TokenProvider and QueueClient. Reflection causes the following problems:

  • You must create derived types from these abstract types that expose all their required overrides.

  • You must expose the internal types that real versions of these classes actually rely on.

  • For the internal types, you must recreate their constructors or factory methods in clever ways to excise the dependency on the real Service Bus.

The better option is to insert your own helper type. That is all you need to mock, detour, and isolate the code from the real-world Service Bus.

Enabling Code Coverage

To analyze what these unit tests verified, we can examine code coverage data. If we run both unit tests using MS Test, we see that they pass. We also see related run details from the Test Explorer dialog.

Figure 2

Figure 2

You can run code coverage for your tests from Test Explorer. Right-click the test, and select Analyze Code Coverage for selected Tests. The results will appear in the Code Coverage Results window. To activate data collection for code coverage, configure the Local.testsettings file under the Solution Items. Opening this file starts the Test Settings editor.

If you have a solution that does not include the Local.testsettings file, add it to the solution using the following procedure.

  1. Click the Add New Item button.

  2. Select Test, and then click Test Settings.

    Figure 3
    Figure 3

  3. Click the Data and Diagnostics tab, and select the check box to the right of the Code Coverage row.

  4. Next, click the Configure button Figure A.

  5. In the Code Coverage Detail window, select all the assemblies that you will test and click OK.

    Figure 4
    Figure 4

  6. To dismiss the Test Settings editor, click Apply and Close.

  7. Rerun your tests, and click the Code Coverage button. Your Code Coverage Results should look like Figure 5.

    Figure 5
    Figure 5

  8. Click the Show Code Coverage Coloring icon, and then navigate down to a method in the grid.

  9. Double-click the method. Its source code will have a color that indicates the areas that were tested. A green color indicates that the code was tested. Gray indicates that the code was partially tested or untested.

    Figure 6
    Figure 6

Using Code Digger to Enhance Code Coverage

Manually creating unit tests is valuable, but Code Digger can also help you intelligently upgrade your unit tests. It does this by trying parameter values you may not have considered. After installing Code Digger, you can explore a method by right-clicking the method in the code editor and selecting Generate Inputs/Outputs Table.

Figure 7

Figure 7

After a short while, Code Digger will finish with its processing and the results will stabilize. For example, Figure 8 shows the results of running Code Digger on the Worker Role's TryProcessOrder method. Notice that Code Digger was able to create one test that resulted in an exception. Code Digger also shows the inputs that it created to generate that exception (a null value for the queue parameter), which is even more important.

Figure 8

Figure 8

Useful Links

Was this page helpful?
(1500 characters remaining)
Thank you for your feedback

Community Additions

ADD
Show:
© 2014 Microsoft