Guidance for Build, Deploy and Test Workflows

This topic describes the recommended approaches for your build, deploy, and test workflow. You can determine the best approach to use based on your needs. However, your needs might change during the phases of your project cycle. At the beginning of your project, you might just build your application on a nightly basis. As your project progresses, you might want to test the quality of your build by adding unit tests to this workflow. These are often called build verification tests or smoke tests. When your test teams are ready to run tests, you might want to automate the workflow to include deploying your application. You can then run manual or automated tests on the latest version of the application that you have deployed. Or, you might want to build, deploy and run automated tests as your complete workflow to verify the quality of your build when it is deployed.

You can run automated tests as part of these workflows from the assembly for a test project using mstest.exe. Alternatively, you can run automated tests from a test suite in your test plan using tcm.exe. To run automated tests from a test plan, you must do these tasks:

1. Create a test plan and test suite: Defining Your Testing Effort Using Test Plans.

2. Associate your automated tests with test cases in a test suite: How to: Associate an Automated Test with a Test Case.

3. Create a physical or virtual environment: Environments.

Use the information in the following sections to set up your software components for the tasks that you need in your workflow:

• Requirements

  The following section describes the requirements to use build, deploy and test as part of your workflow: Requirements.

• Build

  If you just want to build your application, you can use the default build template to do that. For more information about how to set up a build, see Building the Application.

• Build and Test

  If you want to run automated tests as part of your build process, you can select the tests to run from your test project by using the default build template. If the tests pass, then you consider that build is successful. Build and Run Automated Tests

  Important

  This runs tests as part of the build process by using mstest.exe. For more information about mstest.exe, see MSTest.exe Command-Line Options.

• Build and Deploy

  If want to run manual tests from your test plan using the latest build of your application, you can build and then deploy your application to a physical or virtual environment. Build, Deploy and Run Manual Tests from a Test Plan

• Build and Deploy and Test

  If you want to check the quality of the application after deployment, you can build, deploy your application and run automated tests from a test plan using a physical or virtual environment. Build, Deploy and Run Automated Tests from a Test Plan Using an Environment

  Important

  This runs tests as part of the build process by using tcm.exe. For more information about tcm.exe, see tcm: Importing and Running Automated Tests for a Test Plan from the Command Line.

Requirements

Based on the tasks that you want in your workflow, you can install the required components for each task on physical or virtual machines. The following illustration shows an example of how you might choose to install your software based on the information in this section:

Software Requirements for All Tasks

• Team Foundation Server

Software Requirements for Build

• Build controller

  At least 1 build controller is required for each team project collection

• Build agent

  At least 1 build agent is required for each build controller

Software Requirements to Deploy

• 1 build controller (can use the same build controller that is used to build the application)

• For a virtual environment: 1 build agent and 1 lab agent on each machine in the virtual environment

Software Requirements to Test with Environments

• Test Controller

  At least 1 test controller is required for each team project collection

• Test Agents

  1 test agent is required on each machine in the environment

• For a virtual environment: 1 lab agent on each machine in the virtual environment (in addition to the test agent)

For more information about how these components work together to run tests, see Setting Up Test Machines to Run Tests or Collect Data.

For more information about how to install these components, see Installing and Configuring Visual Studio Agents and Test and Build Controllers.

The following sections can help you decide on which machines to install your software components. Then you can determine how many machines you will need for your setup.

Machine Requirements for Build Controllers and Build Agents

To determine the machines that you require, you must base your topology on the following information:

• You must have one build controller for each team project collection.

• You can only have one build controller on each computer.

• In a typical installation, the tasks that the build agent performs can place heavy demands on the processor. This could significantly decrease the performance of your Team Foundation Server. In this case, you will want to put your build agent on a separate computer from your Team Foundation Server.

• You can install the build controller on the same computer as your Team Foundation Server. If the build controller manages many active build agents, this can use more of the system’s memory and you might decide to install the build controller on a separate computer.

For example if you have three team project collections and each one has applications that must be built, you need three computers for the three build controllers or two extra computers if you install one build controller on your Team Foundation Server.

Machine Requirements for Test Controllers and Test Agents

To reduce communication issues between software components, we recommend that the test controller machine and Team Foundation Server machine are in the same domain. For more information about how to install components in different domains or in workgroups, see Requirements for Workgroups and Multiple Domains.

To determine the machines that you require for test controllers, you must decide on the topology that you require based on the following information:

• If you are testing with a physical or virtual environment, you need a test controller.

• You can use a single test controller with multiple physical or virtual environments.

• You must register a test controller with the team project collection that contains your environments.

• Each test controller can only be registered with a single team project collection.

• Each test controller that you require must be installed on a separate machine.

For example, if you have three team project collections for which you want to create environments, you must have at least three test controllers. This requires at least three machines.

Reducing Machines Required For Build Controllers and Test Controllers

If you need multiple build controllers and test controllers, you can reduce your machine requirements by installing a test controller on the same machine as a build controller. For example, if you have three team project collections A, B and C and each one requires a build controller and test controller, you can install the build controller and test controller on the same machine. By using this approach, you would only need three machines for the controllers instead of six machines.

Build and Run Automated Tests

To use build and test as your workflow tasks, you must provide the following information in the default build definition that you create:

• The test assembly that contains your tests

• (Optional) The test categories to use if you want to select specific tests from the assemblies

• The test settings to use to run your tests

  For more information about test settings, see Setting Up Machines and Collecting Diagnostic Information Using Test Settings.

The build agent uses mstest.exe to run the tests from the test assembly that is located in the drop folder as shown in the following illustration. You can use the default test settings (local.testsettings) from the solution for your application to run your tests locally using the build agent.

You cannot collect information by using diagnostic data adapters when you run tests with just a build agent. If you want to use diagnostic data adapters with this scenario, you must install a test agent on the same machine as your build agent. Alternatively, you can use a physical or virtual environment, as described in the following section: Build, Deploy and Run Automated Tests from a Test Plan.

For the detailed steps about how to build and run tests for your workflow, see How to: Configure and Run Scheduled Tests After Building Your Application.

If you are just learning about how to use Visual Studio, Team Foundation Server and Team Foundation Build to define a workflow, you can install all the components on one machine. But we do not recommend this when you have more users of this system and you are running multiple builds.

Build and Deploy

To build and deploy your application for your workflow, you must use a physical or virtual environment.

Virtual Environments

To use a virtual environment that uses Visual Studio Lab Management to build and deploy, you use the lab default template. This lab template enables you to do the following actions:

• Select the virtual environment that you want to use

• Select a snapshot for that environment to use as your starting point for deployment

• Select the build definition or build to use to deploy your application

• Add scripts to run to deploy the application from the drop folder

• Take a snapshot of the virtual environment after deployment as part of the build and deploy workflow

You can use the virtual environment to run manual tests using the build that was deployed onto this environment or you can run automated tests. For more information about how to build and deploy an application to a virtual environment, see How to: Deploy an Application on a Virtual Environment. For more information about how to run manual tests using a virtual environment, see How to: Run Manual Tests and Create Reproducible Bugs With Virtual Environments.

Physical Environments

If you want to deploy your application to a physical environment, you can customize the template for a build definition. For more information about how to customize a build process template, see Create and Work with a Custom Build Process Template.

You can use this environment to run manual tests or automated tests from your test plan. For more information about how to run these tests, see Running Manual Tests Using Test Runner and Essential Guide for Running Automated Tests from a Test Plan.

Build, Deploy and Run Automated Tests from a Test Plan using an Environment

To run automated tests that are part of a test plan in your build, deploy and test process, you must use a physical or virtual environment. The build definition uses tcm.exe to run your tests. When you run these tests as part of your workflow, a test run is created for these tests and the test results can be reviewed and analyzed using Microsoft Test Manager. For more information about how to analyze your test runs, see How to: Analyze Test Runs Using Microsoft Test Manager. This also enables you to view historic data about the quality of your builds. You can determine when there are consistent issues with the quality of the builds and which areas of the application have tests that fail.

You can use physical or virtual machines when you create a physical environment for testing, or you can use Visual Studio Lab Management to create a virtual environment. A virtual environment enables you to deploy your application by using each machine in the environment in a known state from an existing snapshot. In addition, you can take snapshots of the environment after the application is deployed so that you can return to this known state for testing a bug or running manual tests. A virtual environment gives you much more flexibility. For more information about how to create these environments, see Creating a Physical Environment to Use for Testing and Creating Virtual Environments.

Virtual Environments

To deploy your application to a virtual environment, you can use the default lab template definition that is provided with Team Foundation Build. To use a virtual environment, you must have Visual Studio Lab Management. This lab template enables you to do the following actions:

• Select the virtual environment that you want to use

• Select a snapshot for that environment to use as your starting point for deployment

• Select the build definition or build to use to deploy your application

• Add scripts to run to deploy the application from the drop folder

• Select the test suites, test configuration and test settings for your test run

• Take a snapshot of the virtual environment after deployment as part of the build and deploy workflow

For more information about how to set up your build definition for the default lab template to build, deploy and test with a virtual environment, see How to: Configure and Run Scheduled Tests After Building and Deploying Your Application.

Physical Environments

To deploy your application to a physical environment and run automated tests, you can customize the template for a build definition. For more information about how to customize a build process template, see Create and Work with a Custom Build Process Template.

See Also

Concepts

Customizing Lab Management Workflow

Setting Up Test Machines to Run Tests or Collect Data