Streamline Testing Process with Test Impact Analysis

Microsoft Corporation

Authors

Daryush Laqab – Visual Studio Program Manager

Tim Varcak – Software Design Engineer in Test

Date

May 2010

Summary

Software systems have (or should have) a comprehensive set of tests, including unit tests, automated tests, and manual tests. As developers work on the code, the application has to remain under constant scrutiny to ensure it still does what it is supposed to do. That is where testing comes into play. Developers run their unit tests to make sure their code still functions as it is intended, while the QA group tests the application to verify the accuracy, reliability and performance of the application. But, it is hard to know which tests are actually needed to be re-verified as a result of a code change, so it is common to test the entire area where the code change occurred. Until now. Test Impact Analysis helps the developer run only the impacted automated tests and enables QA to skip the tests that were not impacted.

Applies To

Visual Studio 2010

Contents

Introduction to Test Impact Analysis

How does Test Impact Analysis help Developers?

Enabling Test Impact Collection

Test Impact View

How does Test Impact Analysis help software Testers?

Make Test Impact Analysis part of the Process

Introduction to Test Impact Analysis

Today, there is no mechanism by which we can look at a code change and identify the tests that need verification. Knowing which tests to verify has been an imprecise process (at best), relying on extensive knowledge of both the code base and existing tests. As a result, after a code change, developers either verified too many unit tests or too few unit tests, resulting in either reduced productivity or reduced quality. QA faces a similar situation: they either verify too many tests (automated or manual), or too few, again resulting in lower productivity or lower quality. Test Impact Analysis is changing this undesirable reality.

This white paper describes Test Impact Analysis, one of the new features in Visual Studio 2010. As its name implies, Test Impact Analysis (TIA) helps in analysis of impact of development on existing tests. Using TIA, developers know exactly which tests need to be verified as a result of their code change. They can then verify the unit tests directly from inside Visual Studio. Likewise, Testers know exactly which tests they need to verify for each build.

The following sections describe the benefits of TIA for developers and testers in more details.

How does Test Impact Analysis help Developers?

Before we get deep into how developers use TIA, it is helpful to talk a bit about how Test Impact data is actually collected.

Collecting Test Impact Analysis Data

Figure 1: Test Impact Analysis within the Development Process

Identifying impacted tests is fundamentally a comparison task of comparing the current build to the previous build. Based on the comparison, a set of methods are identified as "modified." Visual Studio will then examine the Test Impact baseline, which correlates methods to tests, and based on that information, Visual Studio will display the impacted tests.

There are two areas that merit further discussion:

Intermediate Language Differential (IL Diff)

In order to produce an accurate list of impacted tests, Visual Studio must detect an impactful change in the code. As such, once a local build occurs (resulting in new IL), Test Impact engine is going to perform an IL Diff between the newly created IL and the existing IL. Should it find any IL difference in any method, it will flag that method as "modified." Once a list of modified methods is in place, Visual Studio determines the list of tests calling into that method and flags those tests as "impacted." The Test Impact engine will look in two different locations to find tests calling into the modified method. One of those locations is Team Foundation Server (more on this later), and the other is the local Test Impact data store, which brings us to Test Impact Baseline.

Test Impact Baseline Data

As mentioned above, when a meaningful change in code is determined through IL Diff, Visual Studio looks into the baseline data to identify which tests call into the modified methods. The Test Impact engine updates its baseline as tests are successfully executed (to eliminate the possibility inaccurate Test Impact information). As such, it is important to create and maintain a comprehensive baseline.

Enabling Test Impact Collection

Test Impact collection is enabled in testsettings file. Visual Studio 2010 will add two testsettings file to each solution (each solution which has a Test Project, of course): one is called Local.testsettings and the other is called TraceandTestImpact.testsettings. From TIA’s perspective, the only difference is that TraceandTestImpact.testsettings has Test Impact collection enabled while Local.testsettings does not. One can, however, enable Test Impact collection in any testsettings file. The Test Impact collection option is located in Data and Diagnostics tab of the testsettings file:

Figure 2: Enabling Test Impact Collection

Once you have enabled Test Impact collection, you can further configure the collection by clicking on the "Configure" button. In particular, you can specify the modules and/or processes to be included for (or excluded from) Test Impact collection. If you application is an ASP.NET application running under IIS, make sure the "ASP.NET data collection from IIS" check box is enabled in the Advanced tab. Figures 3, 4, and 5 below depict Test Impact collection configuration options:

Figure 3: Configuring Test Impact Modules

Figure 4: Configuring Test Impact Processes

Figure 5: Enabling Test Impact Collection for ASP.NET applications running under IIS

Test Impact View

Test Impact View is where Visual Studio shows the impacted tests. It can be found in Test, Windows, Test Impact View:

Figure 6: Finding Test Impact View

Recall that the Test Impact engine performs an IL Diff between the new build and existing build to identify the "modified" methods. The Test Impact engine will then cross reference this list with list of tests known to call into the modified methods. The Test Impact engine performs this cross reference against two data sources: one is the local Test Impact data store, in which Test Impact engine will find the list of automated tests (for example, Unit Tests) known to call into the modified methods.

If there are no Team Foundation Server connections configured for Visual Studio, the Test Impact engine populates the Test Impact View with the list of impacted automated tests. However, if a Team Foundation Server connection is configured for Visual Studio, the Test Impact engine will query the Test Impact data store within Team Foundation Server for a list of impacted tests. The union of the two lists is then used to populate the Test Impact View.

The Test Impact data store within Team Foundation Server houses the list of test cases (for example, manual tests created using Microsoft Test Manager) along with the methods they reference inside the code base. This information is used by Visual Studio to identify which test cases are impacted as a result of a code change. Test cases need to be verified via Microsoft Test Manager and cannot be verified through the Visual Studio IDE. However, test cases can provide the developer with more information about the impact of the code change by showing a more complete view of impacted tests.

Figure 7: TFS test cases shown in Test Impact View

The Test Impact View toolbar enables you to analyze, run, and debug impacted tests. Figure 8 depicts the commands of the Test Impact View toolbar:

Figure 8: Test Impact View's Toolbar

There are three ways in which you can analyze impacted tests in Test Impact View. The first (default) view shows the list of impacted tests, and for each selected impacted test, it lists the code modifications which resulted in the need for the test to be re-verified. Figure 7 above depicts this default view. You can always come back to this view by clicking "Show Impacted Tests" button on Test Impact View’s toolbar.

You can, however, pivot the Test Impact data another way: by grouping the code changes first and then viewing the tests each code change impacted. You achieve this by clicking "Show Code Changes" on Test Impact View’s toolbar:

Figure 9: Test Impact View - Show Code Changes

You do not, however, have to make an explicit code change to view or run the tests impacted for a given method. You can simply right click on the method in the Visual Studio code editor and then choose either "Show Calling Tests" or "Run Calling Tests":

Figure 10: View or run tests calling a method

How does Test Impact Analysis help software Testers?

Developers are not the only beneficiaries of Test Impact Analysis. Microsoft Test Manager also integrates with Test Impact Analysis to highlight the impacted manual tests anytime a new build is assigned to a test plan. Anytime a new build is assigned to a test plan, Test Impact Analysis analyzes the change sets in the new build and recommends a set of tests to be re-verified:

Figure 11: View Impacted Manual Tests in Microsoft Test Manager

As in Visual Studio, Test Impact Analysis must be enabled in Microsoft Test Manager before Test Impact can be analyzed. You enable Test Impact Analysis in a Test Setting, under the Data and Diagnostics tab:

Figure 12: Enable Test Impact Analysis in a Test Setting

Once Test Impact Analysis is enabled, Test Impact data is collected during test runs. Once the test has successfully passed and the tester selects "End Test," Test Impact data is attached to the test results and is published to the corresponding Team Foundation Server. Note that if test does not pass, the Test Impact data is considered inadequate and is discarded because the system cannot be certain that all necessary call paths are covered:

Figure 13: Test Impact data is attached to a successful test

Make Test Impact Analysis Part of the Process

The Team Foundation Server check-in policy has been enhanced in Team Foundation Server 2010 so that the policy can now be applied to impacted tests only rather than to all tests. As a result, the overhead of the policy can be lowered without any impact on the ultimate quality of the application:

Figure 14: Testing Policy applied only to Impacted Tests

By default, each Team Foundation Server 2010 build definition performs Test Impact analysis on both automated and manual tests. When a build is completed and a report is generated, the build report associates impacted tests to change sets, and lists each impacted tests alongside the code changes that resulted in an impact on that test. Using this information, management can have an accurate view of the potential impact of a set of code modifications on the overall quality of the application:

Figure 15: Impacted Tests section within a Build Report

See Also

Tasks

How to: Identify the Test Impact of Code Changes During Development

Troubleshooting Test Impact Analysis Issues

Concepts

Technical Articles for Visual Studio Application Lifecycle Management

Visual Studio Application Lifecycle Management