SALES: 1-800-867-1380
1 out of 3 rated this helpful - Rate this topic

Guidance on Efficiently Testing Azure Solutions

Author: Suren Machiraju

Reviewers: Jaime Alva Bravo and Steve Wilkins

After designing, coding, and deploying your Azure AppFabric solution you sometimes find that it does not work. This article provides guidance on how to test your Azure AppFabric applications through the software development lifecycle. The testing scope includes the business logic and the comprehensive end-to-end scenario tests. The present article demonstrates how to:

  • Develop unit tests for the business logic components while eliminating dependencies on Azure AppFabric components.

  • Design integration end-to-end tests and eliminate overhead for setup, initialize, cleanup, and teardown of the Azure AppFabric Service resources (for example, Queues) for every test run. Further eliminate creating duplicate infrastructures for ACS namespaces, Service Bus Queues, and other resources.

Additionally, this article provides an overview of the various testing technologies and techniques for testing your Azure AppFabric applications.

Updates to This Article

The following changes affect the contents of this article:

  1. Unit Tests are now available in Visual Studio Express 2012.

  2. In Visual Studio 2012, private accessors are deprecated. For more information about private accessors, see How To: Test a Private Method, and How to: Regenerate Private Accessors.

  3. Also, beginning with Visual Studio 2012, the testing of private methods is not recommended. For an explanation, see My Take on Unit Testing Private Methods.

  4. A newer version of Moles is integrated with Visual Studio 2012, and is now known as Microsoft Fakes. This feature is described in Isolating Unit Test Methods with Microsoft Fakes.

  5. In Visual Studio 2012, a new version of Code Coverage is now available and is invoked directly from Test Explorer. For a description, see Visual Studio 2012 RC—What's new in Code Coverage.

The Types of Tests

There are two kinds of tests:

  • Unit Tests are narrowly focused tests designed to exercise a single, specific function. These tests are referred to as the Code Under Test or CUT. Any dependencies that the CUT requires must be removed.

  • Integration Tests are broader tests for exercising multiple bits of functionality at the same time. In many cases, these resemble unit tests that cover multiple feature areas and include multiple dependencies.

Overall, these tests focus on creating and using Test Doubles. We use the following types of Test Doubles:

  • Mocks are simulated objects that imitate the behavior of real objects in controlled ways. Mocks replace objects that are dynamically produced by a framework.

  • Fakes are simulated objects that implement the same interface as the object that they represent. Fakes return predefined responses. A fake contains a set of method stubs and serves as a replacement for what you build by hand.

  • Stubs simulate the behavior of software objects.

Upon executing, these tests can verify both state and behavior. For example, state includes the result after calling a method and returning a specific value. An example of behavior is calling a method in a certain order, or a certain number of times).

The Azure Dependencies

One of the major goals of unit testing is to eliminate dependencies. For the Azure framework, these dependencies include the following:

  • Service Bus Queues

  • Access Control Service

  • Cache

  • Windows Azure Tables, Blobs, Queues

  • Windows Azure SQL Database

  • Windows Azure Drive (formerly Cloud Drive)

  • Other Web Services

When building tests for Azure Applications we replace these dependencies to focus the tests on exercising the logic.

The examples of Service Bus Queues (including tools and techniques) that are provided here also apply to all other dependencies.

The Testing Framework

To implement the testing framework for your Azure AppFabric applications, you need:

  • A unit testing framework to define and run your tests.

  • A mocking framework to help you isolate dependencies and build narrowly scoped unit tests.

  • Tools to help with automatic unit test generation for increased code coverage.

  • Other frameworks that can help with testable designs by taking advantage of dependency injection and applying the Inversion of Control Pattern (IoC).

Unit Testing with MS Test

Visual Studio includes a command-line utility MS Test used to execute unit tests created in Visual Studio. Visual Studio also includes a suite of project and item templates to support testing. You typically create a test project, and then add classes (known as test fixtures) adorned with the [TestClass] attribute. The classes contain methods adorned with the [TestMethod] attribute. Within MS Test, various windows within Visual Studio enable you to execute unit tests defined in the project, and review the results after you execute them.

noteNote
Visual Studio 2010 Express and Test Professional editions do not contain MS Test.

In MS Test, unit tests follow an AAA pattern - Arrange, Act, Assert.

  • Arrange - build up any prerequisite objects, configurations, and all other necessary preconditions and inputs needed by the CUT.

  • Act - perform the actual, narrowly scoped test on the code.

  • Assert - verify that the expected results have occurred.

The MS Test framework libraries include the PrivateObject and PrivateType helper classes. These classes use reflection to make it easy to invoke nonpublic instance members or static members, from within the unit test code.

Premium and ultimate editions of Visual Studio include enhanced unit test tooling. The tools integrate with MS Test. The tools also enable you to analyze the amount of code exercised by your unit tests, and color code the source code to indicate coverage. The feature is called Code Coverage.

Mocking with Moles

A goal of unit testing is to be able to test in isolation. However, often the code under test cannot be tested in isolation. Sometimes the code is not written for testability. To rewrite the code would be difficult because it relies on other libraries that are not easily isolated. For example, code that interacts with external environments is not easily isolated. A mocking framework helps you isolate both types of dependencies.

A list of mocking frameworks you might consider is available in the links section at the end of this article. This article focuses on how to use Moles, a mocking framework from Microsoft Research.

Most mocking frameworks create mock types dynamically by deriving from the type you indicate for the members you want to change. Few frameworks support the handling of sealed classes, nonvirtual methods, static members, or delegates. Events that require alternative techniques (such as using the .NET Profiling API) are also rarely covered. Moles framework provides this advanced support.

Most of the mocking frameworks do not enable you to rewrite the logic within a constructor. Imagine creating an instance of a type that you have to isolate. The constructor takes dependencies you want to isolate. For that, you must creatively mock the calls made by the constructor, instead of the constructor itself. Moles can be used to insert logic before constructor calls, but it stops short of letting you completely rewrite the constructor’s logic.

The Moles framework provides all these features at no cost to you. In addition, Moles integrates well with Visual Studio 2010 and MS Test. And tests generated with Moles are fully debuggable within Visual Studio.

Automated Test Creation with PEX

PEX (Program EXploration) is a white-box test generation tool available to MSDN subscribers from Microsoft Research. You can use PEX to create unit tests automatically. PEX results in high code coverage by intelligently selecting input parameters for CUT, and recursively for parameterized unit tests. PEX can build the smallest number of tests that produce the greatest code coverage.

PEX installs and runs fully integrated within Visual Studio 2010, and installs Moles as part of the installation process.

Frameworks for Dependency Injection and Inversion of Control (IoC) Containers

You can use Microsoft Unity for extensible dependency injection and IoC. It supports interception, constructor injection, property injection, and method call injection.

Microsoft Unity and similar tools help you build testable designs that let you insert your dependencies at all levels of your application. (It is assumed that your application was designed and built with dependency injection, and one of these frameworks, in mind.)

These frameworks are great for writing testable code, and ultimately good code, but can be demanding in their upfront design requirements. DI and IoC containers are not covered in this article.

Testing Azure Solutions

In this section, we describe a solution that includes a website hosted on a web role. The website pushes messages to a Queue. Then a worker role processes messages from the Queue. We are interested in testing aspects of all three.

Unit Testing Controller Actions in a Web Role

Imagine you have a website that creates orders, and these orders are queued for processing by using a Service Bus Queue. The webpage resembles the following figure:

TestingAzureSolutions1

When the user clicks Create, it posts the new order to the Create action on the associate controller. The action is implemented as follows:

noteNote
The AppFabricQueue class here is a wrapper class that uses the Service Bus .NET APIs (such as MessageSender and MessageReceiver) to interact with Service Bus Queues.

[HttpPost]
public ActionResult Create(Order order)
{
    try
    {
        string qName = RoleEnvironment.GetConfigurationSettingValue("QueueName");
        string qNamespace = RoleEnvironment.GetConfigurationSettingValue("QueueServiceNamespace");
        string qUsername = RoleEnvironment.GetConfigurationSettingValue("QueueUsername");
        string qPassword = RoleEnvironment.GetConfigurationSettingValue("QueuePassword");        

        //Push order to Service Bus Queue
        AppFabricQueue queue = new AppFabricQueue(new QueueSettings(qNamespace, qName, Username, qPassword));
                
        queue.Send(order);

        queue.CloseConnections();
  
        return View("OrderCreated");
    }
    catch (Exception ex)
    {
        Trace.TraceError(ex.Message);
        return View("Error");
    }
}

Notice that the code retrieves settings from configuration through RoleEnvironment and sends a message that contains the order to the queue.

Here is how to build a unit test for this method by using Moles. Notice that the Create action uses the following methods:

  • GetConfigurationSettingsValue (RoleEnvironment class)

  • Send (AppFabricQueue class)

We want to build detours for these methods by using Moles to control their behavior and remove the dependencies on the real environment. Controlling behavior of these methods and removing their dependencies on the real environment eliminates the need to run the tests in the Azure emulator, or to call out to the Service Bus Queue. We exercise the Create action on the controller to verify that the Order input is the one sent to the queue. The method checks that the input has the Order Id and Description as input to the action. It then checks that the OrderCreated view is displayed as a result.

Accomplishing this with Moles is easy. Within the Test project, right click the assembly that contains the types you want to mock; then select Add Moles Assembly.

In the example, we select Microsoft.Windows.Azure.ServiceRuntime and select Add Moles Assembly. An XML file named Microsoft.WindowsAzure.ServiceRuntime.Moles is added to the test project. Repeat the action for the AF.Queues assembly.

When you build the Test project, references are added to the autogenerated mocked versions of these assemblies.

In the example the generated assemblies are Microsoft.Windows.Azure.ServiceRuntime.Moles and AF.Queues.Moles.

Create a unit test method and apply the [HostType("Moles")] attribute. Within the Unit Test, you use the "moled" types by accessing them through a Moles namespace. All moled types start with an "M."

For example, the moled type for AppFabricQueue is accessed through AF.Queues.Moles.MAppFabricQueue.

You define your detours (the alternate logic to execute) by assigning a delegate to the method you want to replace.

For example, to detour RoleEnvironment.GetConfigurationSetting, define a delegate for MRoleEnvironment.GetConfigurationSettingValueString.

The complete unit test is as follows. Observe the detours provided for:

  • RoleEnvironment.GetConfigurationSetting

  • QueueSettings constructor

  • Send and CloseConnection methods of the AppFabricQueue class

[TestMethod]
[HostType("Moles")]
public void Test_Home_CreateOrder()
{
    //Arrange
    Microsoft.WindowsAzure.ServiceRuntime.Moles.MRoleEnvironment.GetConfigurationSettingValueString = (key) =>
    {
        return "mockedSettingValue";
    };

    bool wasQueueConstructorCalled = false;
    AF.Queues.Moles.MAppFabricQueue.ConstructorQueueSettings = (queue, settings) =>
    {
        wasQueueConstructorCalled = true;
    };

    Order orderSent = null;
    AF.Queues.Moles.MAppFabricQueue.AllInstances.SendObject = (queue, order) =>
    {
       orderSent = order as Order;
    };

    bool wasConnectionClosedCalled = false;
    AF.Queues.Moles.MAppFabricQueue.AllInstances.CloseConnections = (queue) =>
    {
       wasConnectionClosedCalled = true;
    };

    Order inputOrder = new Order()
    {
       OrderId = "Test123",
       Description = "A mock order"
    };

    HomeController controller = new HomeController();

    //Act
    ViewResult result = controller.Create(inputOrder) as ViewResult;

    //Assert
    Assert.IsTrue(wasConnectionClosedCalled);
    Assert.IsTrue(wasQueueConstructorCalled);
    Assert.AreEqual("OrderCreated", result.ViewName);
    Assert.IsNotNull(orderSent);
    Assert.AreEqual(inputOrder.OrderId, orderSent.OrderId);
    Assert.AreEqual(inputOrder.Description, orderSent.Description);

}

Unit Testing Worker Roles that use AppFabric Queues

The web role takes care of adding an order to the queue. Now consider how to test a worker role that would process the orders by retrieving them from the queue. The core item to test in the worker role is defined in the implementation of RoleEntryPoint, typically in the OnStart and Run methods.

The Run method in our worker role periodically polls the queue for orders and processes them.

public override void Run()
{
        AppFabricQueue queue = null;

        try
        {
                string qName = RoleEnvironment.GetConfigurationSettingValue("QueueName");
                string qNamespace = RoleEnvironment.GetConfigurationSettingValue("QueueServiceNamespace");
                string qUsername = RoleEnvironment.GetConfigurationSettingValue("QueueUsername");
                string qPassword = RoleEnvironment.GetConfigurationSettingValue("QueuePassword");

                queue = new AppFabricQueue(new QueueSettings(qNamespace, qName, qUsername, qPassword));
                queue.CreateQueueIfNotExists();

                while (true)
                {
                    Thread.Sleep(2000);

                    //Retrieve order from AppFabric Queue  
                    TryProcessOrder(queue);
                }
        }
        catch (Exception ex)
        {
                if(queue !=null)
                    queue.CloseConnections();
                
                System.Diagnostics.Trace.TraceError(ex.Message);
        }

}

We want a test to verify that the routine correctly retrieves a message.

In this case, we add a moled assembly for the worker role. For this specific project, the Mvc3Web assembly is moled because the RoleEntryPoint is hosted there. The following is the complete unit test for run method of the worker role.

[TestMethod]
[HostType("Moles")]
public void Test_WorkerRole_Run()
{
    //Arrange
    Microsoft.WindowsAzure.ServiceRuntime.Moles.MRoleEnvironment.GetConfigurationSettingValueString = (key) =>
    {
        return "mockedSettingValue";
    };

    bool wasQueueConstructorCalled = false;
    AF.Queues.Moles.MAppFabricQueue.ConstructorQueueSettings = (queue, settings) =>
    {
        wasQueueConstructorCalled = true;
    };

    bool wasEnsureQueueExistsCalled = false;
    int numCallsToEnsureQueueExists = 0;
    AF.Queues.Moles.MAppFabricQueue.AllInstances.CreateQueueIfNotExists = (queue) =>
    {
        wasEnsureQueueExistsCalled = true;
        numCallsToEnsureQueueExists++;
    };

    bool wasTryProcessOrderCalled = false;
    int numCallsToTryProcessOrder = 0;
    Mvc3Web.Worker.Moles.MWorkerRole.AllInstances.TryProcessOrderAppFabricQueue =
    (actualWorkerRole, queue) =>
    {
         wasTryProcessOrderCalled = true;
         numCallsToTryProcessOrder++;

         if (numCallsToTryProcessOrder > 3) throw new Exception("Aborting Run");
    };

    bool wasConnectionClosedCalled = false;
    AF.Queues.Moles.MAppFabricQueue.AllInstances.CloseConnections = (queue) =>
    {
         wasConnectionClosedCalled = true;
    };

    Mvc3Web.Worker.WorkerRole workerRole = new Worker.WorkerRole();

            //Act
    workerRole.Run();

            //Assert
    Assert.IsTrue(wasConnectionClosedCalled);
    Assert.IsTrue(wasQueueConstructorCalled);
    Assert.IsTrue(wasEnsureQueueExistsCalled);
    Assert.IsTrue(wasTryProcessOrderCalled);
    Assert.AreEqual(1, numCallsToEnsureQueueExists);
    Assert.IsTrue(numCallsToTryProcessOrder > 0);
}

Now the focus is on detours for the instance methods of the WorkerRole class. You should set the delegate of the AllInstances property of the generated mole type. By using the delegate, any instance you create of the actual type detour through any of the methods for which you have defined delegates.

In the example, we want to use the original instance’s Run method, but provide detours for the instance methods CreateQueueIfNotExists and TryProcessOrder. (This is in addition to the static method detours already discussed for RoleEnvironment.GetConfigurationSettings.) In the code, an exception is thrown so that the infinite loop maintained within Run is exited at a predetermined time.

You may ask, why not just use the MessageSender/MessageReceiver and related classes from the Service Bus SDK directly instead of injecting a helper type? In order to completely isolate the code from calling the real-world Service Bus, there are two choices: (1) write fakes that inherit from the abstract classes in the Microsoft.ServiceBus namespace, or (2) let moles create mock types for them all. The problem with either approach is complexity. With both approaches, you ultimately find yourself reflecting into classes like TokenProvider and QueueClient. Reflection causes the following problems:

  • You must create derived types from these abstract types that expose all their required overrides.

  • You must expose the internal types that real versions of these classes actually rely on.

  • For the internal types, you must recreate their constructors or factory methods in clever ways to excise the dependency on the real Service Bus.

Or you can insert your own helper type. That is all that is needed to mock and detour, and isolate the code from the real-world Service Bus.

Enabling Code Coverage

To analyze what these unit tests tested, we can examine code coverage data. If we run both of the unit tests with MS Test, we see that they pass, and we also see related run details from the Test Results dialog.

TestingAzureSolutions2

By clicking the Show Code Coverage Results button (the rightmost button near the top of the Test Results window), we see the code covered by all unit tests executed in the run. However, at this stage, there is no data. Instead, there would be a message that indicates no coverage data could be found. We first have to enable collection of code coverage data before running the tests.

To enable data collection for code coverage, configure the Local.testsettings file under the Solution Items. Opening this file brings up the Test Settings editor.

noteNote
If you have a solution that does not have Local.testsettings file, add it to the solution using the Add New Item command. Select Test, and then Test Settings.

TestingAzureSolutions3

Click the Data and Diagnostics tab, and the check the box to the right of the Code Coverage row. Next, click the Configure button (the one just above the grid that contains Code Coverage). In the Code Coverage Detail window, select all your assemblies under test and click OK.

TestingAzureSolutions4

Then click Apply and Close to dismiss the Test Settings editor. Rerun your tests and press the code coverage button. Your Code Coverage Results resembles the following:

TestingAzureSolutions5

Click the Show Code Coverage Coloring icon, and then navigate down to a method in the grid. Double-click the method to see its source code colored to reflect areas that were tested. The following figure shows green for tested code, and gray for partially tested or untested.

TestingAzureSolutions6

Using PEX to Enhance Code Coverage

Manually creating these unit tests is valuable, but PEX can also help you intelligently upgrade your unit tests. It does this by trying parameter values you may not have considered. After installing PEX, you can have it explore a method simply by right clicking the method in the code editor, and selecting Run PEX.

TestingAzureSolutions7

After a short while, PEX will finish with its processing and the results will stabilize. For example, here is the result after running PEX on the worker role's TryProcessOrder method. Notice that PEX was able to create one test that resulted in an exception. PEX also shows the inputs that it crafted to generate that exception (a null value for the queue parameter). Even more important, in the details pane you can see the code that PEX executed. The PEX generated tests can be added to your project simply by selecting them and clicking Promote.

TestingAzureSolutions8

Useful Links


Build Date:

2013-01-07
Did you find this helpful?
(1500 characters remaining)
Thank you for your feedback

Community Additions

ADD
Show:
© 2014 Microsoft. All rights reserved.