Walkthrough: Using Code Coverage in Microsoft Test Manager

Visual Studio 2012

When you run manual tests using Microsoft Test Manager with Visual Studio 2012.1 or later, you can collect code coverage metrics for ASP.NET applications running on Internet Information Server. This walkthrough shows you how to collect code coverage metrics for manual tests run with test runner. You can follow similar steps for automated tests and exploratory tests as well.

This walkthrough contains the following sections:

  • Microsoft Test Manager is included in Visual Studio Ultimate, Visual Studio Premium, or Visual Studio Test Professional

  • Visual Studio 2012.1 or later.

  • Team Foundation Server 2012 Update 1 or later.

  • Test Controller 2012 Update 1 or later.

  • Test Agent 2012 Update 1 or later.

  • To collect code coverage from a web server, you need to add the IIS machine as environment with Web Server role in Lab Manager. See Creating a standard environment.

To collect code coverage metrics, you must first enable the code coverage data diagnostics adapter in your test settings.

  1. Open Microsoft Test Manager.

    Note Note

    To display the Microsoft Test Manager window, choose Start, and then choose All Programs. Point to Microsoft Visual Studio 2012. Open the shortcut menu for Microsoft Test Manager and then choose Run as administrator.

  2. Connect to your team project.

    Tip Tip

    If you are already connected to a team project and have selected a test plan, you can skip this step.

    When you start Microsoft Test Manager for the first time, you are not connected to any team project. You must first connect to Team Foundation Server, select your project collection, and your team project. The team project is the same one that is used by other users on your team to create requirements or user stories, maintain source code, and build the applications that you want to test. Your testing artifacts are created and managed in this same team project.

    The following illustration shows how to connect to your team project.

    Connect to Your Team Project
    Note Note

    Only team projects that you have permission to access will be displayed. For more information about permissions, see Team Foundation Server Permissions.

    Then you must create or select a test plan, as shown in the following illustration. You have to use a test plan in your team project for planning and running your tests.

    Set the Plan Context

    For more information, see Quick Start Guide for Manual Testing using Microsoft Test Manager.

  3. Choose the down-arrow on the center group switcher and then choose Testing Center.

  4. On the center group menu bar, choose Plan and then choose Properties.

    The properties for the currently selected test plan are displayed.

  5. Choose the drop-down arrow on Test settings under Manual runs and then choose <New…>

    The New Test Settings page is displayed with the General page selected.

    Note Note

    You can also create test settings in Lab Center by choosing Test Settings on the center group menu bar, and then choosing New.

  6. Under Name, type the name for the test settings.

  7. (Optional) Under Description, type a description for the test setting so that other team members know its intended use.

  8. Under What type of tests do you want to run, select Manual and then choose Next.

    The New Test Settings page is displayed with the Roles page selected.

    Test settings Roles page
  9. Choose Web Server role.

    Note Note
    • If your test plan does not include any environments with a Web Server role, this role does not display in the Roles step. See Creating a Standard Environment.

    • If at least one environment with Web Server role is not in a Ready state, you will not be able to see any data collectors for this role.

  10. Choose Next.

    The New Test Settings page is displayed with the Data and Diagnostics page selected.

  11. On the Data and Diagnostics page, choose Code Coverage and then choose Configure.

    Configure code coverage

    This opens the Configure Diagnostic Data Adapter dialog box for code coverage.

    Configure adapter
  12. In the Modules tab, you can specify modules to be included for code coverage collection. By default all assemblies (*) loaded by IIS worker process is included. If you want code coverage only on specific assemblies, please specify regular expressions matching your modules. For example: *\\WebAppDAL\.dll.

    Note Note
    • The user account must be a member of the administrator’s security group in order to use the ASP.NET Profiler data and diagnostic adapter. For more information, see Install a Test Agent on the Internet Information Server.

    • Verify that the .PDB files are deployed in the same folder as the Web application under test binaries are located. For a Web application, this will be the publishing folder of the Web application, for example wwwroot\myWebApp.

      If .PDB files are not deployed with application specify additional search paths for symbols. You can configure multiple local or remote paths separated by semicolon. For specifying build numbers in drop folders, use $(Build) macro. For example: C:\inetpub\MyWebApplication\bin;\\vsalm\drops\NightlyBuild\$(Build).

  13. (Optional) In the Advanced tab, you can exclude modules by company name, public key tokens and configure addition symbol paths. If your application under test includes Microsoft assemblies, remove exclusion by company name to be able to get coverage on them.

    Code coverage advanced configuration
  14. Choose Save on the Configure Diagnostic Data Adapter dialog box.

  15. Choose Finish to save the new test settings.

  16. Back in the Properties page, verify that the new test settings is selected in the Test settings drop-down list under Manual runs.

  17. Under Manual runs, verify that the environment with the web server role is selected.

  18. Under Builds, verify that the build you want to use is selected.

Remote collection of ASP.NET Client Proxy for IntelliTrace and Test Impact, IntelliTrace and Test Impact requires manual configuration of elevated permissions. If code coverage is enabled with any of these collectors, log on to the Test Agent machine, open the Test Agent Configuration Tool and configure the Test Agent service to run with Administrator credentials.

If Code Coverage is not enabled with above collectors, this step is not necessary.

Now that you have successfully configured code coverage settings, you can perform manual test runs, In addition, you can collect and track code coverage as a build level metric for your ASP.NET applications. At the end of every test run with code coverage, results are published to Team Foundation Server, where it is processed and tracked as a build level metric.

Note Note

Code Coverage results are processed only for builds queued after applying Team Foundation Server 2012 Update 1. For more information about generating a new build, see Queue a Build.

  1. In Microsoft Test Manager, in the Testing Center group menu bar, choose Test.

  2. In the menu under the Testing Center group menu bar, choose Run Tests if it is not already selected.

  3. In the left pane, under Run Tests, choose either the test plan or the test suite listed under the test plan containing the test that you want to run.

  4. In the list of test cases, choose the test case you want to test and then choose Run.

    Tip Tip

    You can also choose Run with options which allows you to specify the build number, build configuration, test setting and environment to use with the test run.

  5. The tests run with the settings you created in Configure Code Coverage Settings in Your Test Plan.

  6. Choose Start Test.

  7. Launch the application under test after your testing session has started.

  8. To mark the result of a test step, choose the drop-down arrow next to the active icon. The results are shaded with a blue vertical line and appear to the right of the test step number. Then choose either Pass or Fail.

  9. When you have finished your test, choose End Test.

    Selecting test to run in Microsoft Test Manager Start test dialog Running tests with Test Runner
  10. In the toolbar at the top of Test Runner choose Save and Close to save your test results to the Team Foundation Server and close Test Runner.

  1. In the Microsoft Test Manager menu under the Testing Center group menu bar, choose Analyze Test Runs and open your test run. You will find a .coverage file under Attachments. Select the .coverage file and then choose Open.

    The code coverage results file opens in Visual Studio. You can see the various blocks, grouped first by DLLs, and then by namespaces. You can see how many blocks are covered or not covered.

    For more information, see Analyzing code coverage results.

  2. If you want to see what blocks are covered or not covered, in the CODE COVERAGE RESULTS window, double-click the method name. The coloring indicates what blocks are covered and what blocks are not.

    Covde coverage results in Visual Studio

    In addition to viewing the coverage file, the code coverage numbers are also published to the data warehouse for the build that is specified in the test plan properties for the test run. This way, when you open the Build Summary page, you'll get the consolidated code coverage metrics across all test runs for the build.

Issue: Code coverage file is empty and Test run attachments has a collector warning XML with following message: “An error occurred during website instrumentation: Internet Information Services could not be found on the machine“.

It is possible that you have enabled code coverage collection on a machine with no IIS. The most common reason is that you have enabled Code Coverage in the incorrect role.

Workaround: Open test settings and make sure code coverage is enabled and configured on the Web Server role instead of the Local role.

Issue: Code coverage results is empty and the file size is less than or about 10KB, depending on number of blocks of code in application under test.

Collector requires symbols or .PDB files to be able to collect code coverage. Either the symbol files have not been configured or symbols provided do not match instrumented DLLs.

Workaround: Save the coverage file from test run attachment into file system. Analyze coverage file using the following commands:

cd /d "%VS110COMNTOOLS%\..\..\Team Tools\Dynamic Code Coverage Tools"

CodeCoverage.exe analyze /include_skipped_modules /output:analysis.xml <path to file>

Open analysis.xml to determine the reason for empty coverage and take appropriate steps.

Issue: You see data collectors exceptions at the start or end of test runs.

Previous test runs not cleaned up correctly

Workaround: In Services manager restart Visual Studio Lab Agent Service.