Run Tests in Your Build Process

  • Article

You can use Team Foundation Build to run automated tests and analyze the impact of code changes on your tests as part of your build process. For example, you can define a build process to serve as your team’s regularly scheduled build verification test (BVT) run. You can also run automated tests and perform test-related tasks from your custom build processes.

Note

If you want to deploy your application as part of your build process, you must use a build-deploy-test workflow and a lab environment. You can then run automated tests as part of the workflow, or you can run tests separately after the workflow is complete. For more information, see Setting Up Automated Build-Deploy-Test Workflows.

In this topic

  • Prepare to Run Tests in Your Build Process

  • Required Permissions

  • Use a Test Run to Run Automated Tests

  • Choose and Configure a Test Runner

    • Use Visual Studio Test Runner to Run Tests

    • Use MSTest to Run Tests

  • Temporarily Disable Tests

  • Enable Test Impact Analysis

  • Configure a Third-Party Unit Test Framework

  • Run Tests from a Custom Build Process

  • For More Information

Prepare to run tests in your build process

Before you run tests in your build process, you may need to first prepare your tests and your build system.

Prepare Your Tests: Make sure your solution and your test files are checked into version control. See Using Version Control.

Categorize and Prioritize Your Tests (Optional): You can assign categories and priorities to your tests, and then filter on those attributes when you run them in your build. For example, you could create a test category called CI and then specify that category in your continuous integration builds. You could create another category for your build verification tests called bvt and then specify that category in scheduled builds, such as your nightly build. For more information, see Defining Test Categories to Group Your Tests, TestCategoryAttribute, and PriorityAttribute.

Prepare Your Build server: Some kinds of tests can be run only by a build agent on a build server that is specially configured. For example, if you are running coded UI tests, you must configure your build agent to run interactively. Before you try to use your build process to run tests, make sure they can be run on the build server that you plan to use. For more information, see Use Your Build Agent to Run Tests.

Microsoft Visual Studio is required to be installed on the build server for the following scenarios:

  • To build any CPP test project, you must install Visual Studio Professional or later.

  • To run unit tests, or coded UI tests, you must install Visual Studio Professional or later.

  • To use data and diagnostic data adapters:

    1. Code coverage: Visual Studio Premium or later.

    2. Test impact: Visual Studio Ultimate.

    3. IntelliTrace: Visual Studio Ultimate.

  • To build any modern style apps on a build machine: Visual Studio Ultimate or Visual Studio Express for Windows 8 (Operating system on build server must be Windows 8).

  • To compile and run tests for a project with a faked assembly: Visual Studio Ultimate.

Required permissions

To perform these procedures, your Edit Build Definition permission must be set to Allow. For more information, see Team Foundation Server Permissions.

Run automated tests

You can perform one or more test runs in your build that is based on the Default Template. For each run, you can specify the following settings:

  • Which tests are run

  • Which settings are used to run the tests

  • Whether the build should fail if a test fails

To run automated tests in a build process that is based on the Default Template

  1. On the menu bar, choose View, Team Explorer.

  2. In Team Explorer, choose Home iconHome, and then choose Builds IconBuilds.

  3. On the Builds page, choose New Build Definition or open the shortcut menu for the build or build definition you chose, and choose Edit Build Definition.

    The build definition window appears.

  4. On the Process tab of your build definition, expand the Basic node.

  5. Select the Automated Tests check box, and then choose the ellipsis button (...).

    The Automated Tests dialog box appears.

  6. Perform one of the following steps:

    • To add a set of tests, choose Add.

    • To modify a set of tests, choose it, and then choose Edit.

    The Add/Edit Test dialog box appears.

  7. (Optional) Specify the Name of the test run. This name appears in the build results window. If you do not specify a name, the system automatically generates it.

  8. If you want the build to fail if any of the tests in this run fail, then choose Fail build on test failure. If you leave this check box cleared and any test fails, the completed build will be classified as Partially Succeeded.

  9. Choose a test runner. Specify how the test runner will operate using the options that appear. For more information, see Choose and configure a test runner.

Choose and configure a test runner

When you define your build process, choose a test runner that provides the functionality you require.

Capability

Visual Studio Test Runner

MSTest

Run unit tests written in Microsoft.VisualStudio.TestTools.UnitTesting framework.

Yes

Yes

Run coded-UI, generic, and ordered tests

Yes

Yes

Gather code coverage data

Yes

Yes

Run unit tests written in multiple frameworks such as NUnit, xUnit, and others.

Yes

No

Run unit tests in native (for example, C++) binaries.

Yes

No

Run unit tests on a Windows Store app.

Yes

No

Run 64-bit tests

Note

MSTest can run 64-bit tests in binaries compiled with Any CPU specified as the Platform.

Yes

No

Run tests that depend on fakes framework objects.

For more information on using the test fakes framework to create delegate-based test stubs and shims in .NET Framework applications, see Isolating Code under Test with Microsoft Fakes.

Yes

No

Generate database test data and run database unit tests

No

Yes

Gather test impact analysis data

No

Yes

Run load and web tests

No

Yes

Note

Some of these capabilities function only on a build agent on a build server that is specially configured. For more information, see Use Your Build Agent to Run Tests.

After you understand the capabilities that your test run requires, choose a test runner:

  • Visual Studio Test Runner

  • MSTest with test settings file

  • MSTest with metadata file

Use Visual Studio Test Runner to run tests

To use Visual Studio Test Runner to run tests from a build process that is based on the Default Template

  1. In Team Explorer, open Builds, and then add or edit a build definition.

    On the Process page, expand Basic, and select Automated Tests. Choose the ellipsis button (...) at the end of the line and then choose to Add or Edit a test source.

  2. On the Add/Edit Test Run dialog box, under Test runner select Visual Studio Test Runner.

  3. Test assembly file specification

    Specify the binary files that contain the tests that you want to run. Leave the default value (**\*test*.dll) if you want the build agent to search recursively for any .dll files that match *test*.dll in the binaries subdirectory of the build agent's working directory. As an alternative, modify the file specification to meet your needs.

  4. If you want the test run to gather and publish code coverage data, set Options to Enable code coverage.

    Alternatively, you can use the Custom option to specify a .runsettings file. For more information, see Customizing Code Coverage Analysis.

  5. From the Select target platform for test execution menu, choose x86 to test your 32-bit binaries, or x64 to test your 64-bit binaries.

  6. You can specify criteria for the tests that are run.

Specify criteria for the tests run by Visual Studio Test Runner

You can specify name/value pairs to filter the tests that are run. If you use test category and priority attributes to organize and prioritize your tests, then you can filter the tests you are running by using the TestCategory and Priority names.

You can specify test categories in one of the following forms:

  • Specify a single name/value pair to include. For example, you could have a test category that is called bvt. You would set Test case filter to TestCategory=bvt to run only tests in this category.

  • Specify multiple test categories by using the || ("or" operator). For example, you can specify TestCategory=quick||TestCategory=gui to run tests in the quick category and tests in the gui category.

Use MSTest to run tests

To run tests using MSTest in a build process that is based on the Default Template

  1. Use a test run to run automated tests and perform other test tasks, and then on the Add/Edit Test Run dialog box, under Test runner choose:

    • MSTest.exe Runner

      Specify the binary files that contain the tests that you want to run. Leave the default value (**\*test*.dll) if you want the build agent to search recursively for any .dll files that match *test*.dll in the binaries subdirectory of the build agent's working directory. As an alternative, modify the file specification to meet your needs.

      (Optional) Specify a Test Settings File to configure how the tests are run. For more information, see Specifying Test Settings for Visual Studio Tests.

    • MsTest Test Metadata File (deprecated)

      If you choose this option, choose Browse to locate and specify the test metadata file that you want to use. You can then either leave the Run all tests in this VSMDI file check box selected, or you can clear it and then select one or more tests in the Test lists to run list.

      Note

      Test lists are deprecated. For more information, see Defining Test Categories to Group Your Tests.

  2. Choose the Criteria/Arguments tab.

  3. You can filter the tests to run by category.

    You can specify test categories in one of the following forms:

    • Specify a single test category to include or exclude. For example, you could have a test category that is called bvt. You would set this parameter to bvt to run only tests in this category or !bvt to run all tests except those tests in this category.

    • Specify multiple test categories by using the & ("and" operator) and the ! ("not" operator). For example, you can specify quick&gui&!deep to run only those tests in both the quick and gui categories but not in the deep category.

    • Specify multiple test categories by using the | ("or" operator) and the ! ("not" operator). For example, you can specify quick|gui|!deep to run tests in the quick category, tests in the gui category, and any tests that are not in the deep category.

  4. (Optional) Filter the tests to run by priority.

    Tip

    If you assign priorities to your tests, you can use this parameter to better balance thorough testing with faster builds.

    Set Minimum Test Priority to either a positive integer that is equal to or lower than Maximum Test Priority or to -1 to indicate no minimum value.

    Set Maximum Test Priority to either a positive integer that is equal to or higher than Minimum Test Priority or to -1 to indicate no maximum value.

  5. Choose OK.

Temporarily disable tests

If you must temporarily disable tests without deleting the test sets that contain them, expand the Advanced node, and set Disable Tests to True. Set the value back to False when you want to enable tests again.

Enable test impact analysis

Your testers and developers may need to know how code changes that are encompassed in a completed build have affected your tests. When you enable test impact analysis in a build, the system analyzes and then reports on how code changes affected your tests in the build report of the completed build.

To enable test impact analysis in a build process that is based on the Default template

  1. Configure test impact analysis in a test settings file.

    For more information, see How to: Collect Data to Check Which Tests Should be Run After Code Changes.

  2. Create a set of tests that is configured to use the test settings file.

    For more information, see Run Automated Tests earlier in this topic.

  3. Expand the Advanced node, and make sure that Analyze Test Impact is set to True and that Disable Tests is set to False.

Define Multiple Test Runs

You can define as many test runs as you need to meet the requirements of your team's build and test process. For example, you might need to define multiple test runs in a single build in the following scenarios:

  • You want to use the Visual Studio Test runner to test a solution that produces binaries with 32- and 64-bit bitness.

  • You have some tests that can be run by the Visual Studio Test Runner, and other tests that can be run only by MSTest.

  • You have two sets of tests:

    • A set of top-priority core tests that must pass. You define a set of tests that includes a Minimum Test Priority and Maximum Test Priority of 1. You select the Fail build on test failure check box.

    • A set of less important tests that you want to run but that are not required to pass for the build to be usable. You define a set of tests that include a Minimum Test Priority of 2 and a Maximum Test Priority of 3. You leave the Fail build on test failure check box cleared.

  • You want to run the same set of tests with different test settings.

  • You want the main set of assemblies that you build to be subject to code coverage. However, you have another set of assemblies from an external source that do not require code coverage. To enable this kind of process, you could use two sets of tests that are configured to use two groups of test settings files.

Enable a Third-Party Unit Test Framework on a build controller

Your build process can run unit tests based on third-party unit test frameworks only if you have provided your build controller with access to the third-party framework assemblies.

  1. Locate, or if necessary, specify the build controller’s path to custom assemblies.

  2. Locate, or if necessary, create a mapping from the custom assembly folder on the server to a local folder in your workspace.

  3. Obtain a third-party unit test plug-in. You can find links to some of them here: Unit Testing Plugins List.

  4. Rename the plug-in .vsix file to a .zip file. For example, use the command prompt like this:

    C:\Downloads>ren NUnitTestAdapter.vsix NUnitTestAdapter.zip

  5. Unzip contents of the .zip file into the local workspace folder you mapped in Step 2.

  6. Check in the files.

    Tip

    For strategies to work with third-party binary files in version control, see Leverage third-party binaries that your code does not build.

The [Visual Studio 2012.1] includes an enhancement for third-party unit test frameworks to automate their inclusion in team build definitions.

Warning

You may need to install the newest version of the NuGet packages for the 3rd-party unit test framework to ensure the framework includes build definition enhancement.

Enable a Third-Party Unit Test Framework on a build controller - [Visual Studio 2012.1]

  1. In Solution Explorer, open the shortcut menu on the test project and choose Manage NuGet Packages.

  2. In the Manage NuGet Packages dialog box, in the left column, choose Online.

  3. Select the NuGet package for the third-party unit test framework and choose Install.

  4. After the NuGet package is finished installing, choose Close.

  5. In Solution Explorer, open the shortcut menu on the solution and choose Add Solution to Source Control.

  6. You can now queue your build and your tests with the third-party unit test framework will automatically run.

Work with Tests in a Custom Build Process

If your team requires a build process with more deeply customized capabilities, you can run tests and perform other test-related tasks from within your custom build process. For more information, see:

For more information

Define a Build Process that is Based on the Default Template provides more information about how to create a build definition that is based on the Default Template. This topic includes information about Platform bitness settings you can use when compiling your code.

Guidance for Build-Deploy-Test Workflows provides information on deploying an application and running tests in a lab environment using your build process.

Running Tests in Microsoft Test Manager provides more information about how to run tests.

See Also

Concepts

Define Your Build Process

Creating and Managing Tests in Microsoft Test Manager