Specifying Test Settings in Microsoft Test Manager

When you run tests in Microsoft Test Manager, the test framework can collect data such as an action log, a screen and voice recording, or diagnostic trace information for each machine role. You can specify these options in test settings. Test settings control the diagnostic data adapters that actually collect the data.

To test a typical web or distributed system, you will use more than one test machine to perform the roles of client, server, database, and so on. Your test settings specify the roles that are required for your tests, and specify separate diagnostic adapter configurations for each role. When you run your tests in your test plan, a lab environment with the same number of roles is automatically selected. If there are multiple test environments that match the set of roles in your test settings, you can select a different matching environment.

Requirements

• Visual Studio Ultimate or Visual Studio Premium or Visual Studio Test Professional

Do I need to modify the test settings?

When you are first starting to use Microsoft Test Manager, it is easiest not to modify the default test settings.

If you do not specify test settings, the default one will be used, which will capture system information and an action log, which helps in manual or exploratory testing.

-

-

You will have to modify or create a test settings if you want to:

• Add event logs, IntelliTrace or video recording to your test results and bug work items, to help isolate bugs in your application.

• Perform test impact analysis to find out what tests are affected by recent code changes.

• Emulate potential bottlenecks that your app might occasionally encounter in a production environment.

• Configure details of how automated tests cases are run.

How do I do use a test settings file?

In Microsoft Test Manager, in the Properties page of your Test Plan, you can select two test settings files. One is for manual tests and the other is for automated tests. When you run test cases in that plan, these settings are the default choices, but you can override them in individual runs.

You can either create a new test settings file, or select an existing file that has already been defined in your team project. For example if you are creating a new test plan for the next iteration of your team project, you would typically reuse a test settings file that was used in the current iteration.

To select or create a test settings file for your test plan:

Open Testing Center, Plan, Properties. Under Manual Runs or Automated Runs, click the menu at Test Settings and choose either an existing test settings file, or New. To edit the details of an existing test settings file, you can choose Open.

The settings for Automated Runs are used when you run test cases that have been linked to test code. If all your tests are manual, you don’t have to set this option.

To manage the test settings files of your team project:

Open Lab Center, Test Settings. From there you can edit existing test settings files and create new ones.

For additional guidance, see Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox.

Editing test settings

The test settings pages are:

• General

• Roles

• Data and Diagnostics

• Advanced – only for automated tests. For more information see Creating Automated Tests Using Microsoft Test Manager.

• Summary

Test Settings: General

Provide a name for this settings file, and specify whether it is for manual or automated test runs.

Choose Manual to define settings for exploratory tests, test cases that you run manually by following test steps in Microsoft Test Manager, and tests in which you play back a recorded sequence of actions.

Choose Automated to define settings to run test cases that have been associated with test methods in Visual Studio.

Test Settings: Roles

On the Roles page of the test settings, choose a combination of machine roles that is suitable to run your tests. When this test setting is used to run a test, the software under test must be deployed on an environment that has at least the same number of machines, with a matching set of roles. A role is a label such as web server or database server that indicates the intended use of the machine.

The list of available sets of roles is based on the lab environments that are defined in your test project. If you cannot see one that is suitable for your tests, then you must create a new lab environment. For more information, see Creating Lab Environments.

• Roles for manual test runs
  The Local role is always included. This role corresponds to the computer on which you run Microsoft Test Manager and on which you perform the tests.

  You don’t need other roles unless you are testing a distributed or web application and you want to collect diagnostic data from the server machines while you run your tests.

  Tip

  If you have an environment that includes a desktop client, you can run your manual tests on this machine if you install Microsoft Test Manager. Effectively, this machine then becomes the local machine for your test settings because you will run your manual tests on this machine.

  

• Roles for automated test runs
  Automated tests must be run on a lab environment. You must choose a set of roles that includes a role for each machine on which your application is deployed. If it is a simple desktop application, it will require only one role. If it is a distributed application such as a web service, it will also require a role for the web server, and perhaps also for a database server and so forth.

  If your application uses an external service that is not part of your application, you should not include that in the set of roles.

  At Select the role to use to run your automated tests, choose the machine on which the test code will be loaded and run.

  

Test Settings: Data and Diagnostics

On this page you can add and configure diagnostic adapters to collect data for each machine role in your lab environment. In most cases the diagnostic data is included with the test results.

Select each role in turn and check the diagnostic adapters you want to use.

For compatibility information about test settings between Visual Studio 2012 and Visual Studio 2010, see Compatibility of Test Settings with Visual Studio 2010.

Test Settings for Automated tests

These test settings are available only if you are creating a test setting for automated tests. For more information see Creating Automated Tests Using Microsoft Test Manager.

Configuring the Unit Test Add-in

If you automate a test case by linking it to a unit test, configure Unit Test on the Add-ins page of the test settings.

1. For Root folder for the assemblies to be loaded, choose Browse to locate the folder and populate the text box.

   The root folder that is specified can contain environment variables and represents the directory which will be used as the ApplicationBase of the AppDomain that the tests are run in. All the assemblies in this directory will be loadable by your unit tests. In a production environment, a good practice is to set this to the directory where your code under test assemblies are installed. In a development environment, a good practice is to set this to the directory where your code under test assemblies are built to. This makes sure that any references that you have to the product binaries can be loaded and resolved during the discovery and execution of the tests without the need to copy the product binaries around with the tests.

   If no value is set here the ApplicationBase of the AppDomain that the tests are run in is set to the directory which contains the tests.

2. Select or clear the check box for Use the Load Context for assemblies in the test directory.

   By default, most assemblies are loaded into the correct "Load Context" Usually, you should leave Use the Load Context for assemblies in the test directory selected. However, there are some conditions when you might want to turn this off. If there are a large number of assemblies in your test directory, and you have specified a location under Root folder for the assemblies to be loaded, and your tests are not dependent on being loaded in the Load Context, you could see a performance increase if you do not use the Load Context to load these test assemblies. If your tests depend on being loaded in a context other than the Load Context (not typical).

   For more information, see Best Practices for Assembly Loading.

3. Under Folders to use when the tests are run, choose Add folder.

   The Browse For Folder dialog box is displayed.

4. Locate the folder to use and choose OK.

   The Folders to use when the tests are run setting is the setting that you will probably use the most frequently. You can specify multiple paths to folders that assemblies should be resolved from during discovery and execution of the tests. Each of the paths that are specified in this section can contain environment variables. Together with each of the paths that are specified here, there are two options that are associated with it:

   First option   Select the Use Load Context check box to specify that the directory should use the load context when resolving assemblies from the directory (if the load context is not required for the tests to run correctly you might see a performance improvement by clearing this check box).

   Second option   Select the Include sub-folders check box to specify using any sub-folder to include when resolving assemblies from the directory.

5. Under Additional folders to use when discovering tests, choose Add Folder.

   The Browse For Folder dialog box is displayed.

6. Locate the folder to use and choose OK.

   This Additional folders to use when discovering tests is useful when you are either executing the tests remotely under Team Build or doing an automated run from Microsoft Test Manager. The paths provided here will be used for assembly resolution, but only during test discovery. These paths can contain environment variables. When tests are being scheduled to execute remotely from a build drop and not all the dependencies of the test assembly are in the same directory, these paths can be used to make sure that MSTest or the Test Controller can find enough of the dependent assemblies to discover the tests and schedule them to the remote machines for execution.

   For runs being scheduled from Microsoft Test Manager, there is an additional token "%BuildDrop%" that can be used to generically refer to the build drop location. This eliminates the need to create or update a Test Settings each time a new build is being tested. Unfortunately this token is not directly supported through Team Build. However, if the build drop location is set in an environment variable named BuildDrop from the build definition, it will have the same result).

7. Choose Save.

8. Choose Close.

External resources

Guidance

Testing for Continuous Delivery with Visual Studio 2012 – Chapter 3: Lab Environments

Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox

See Also

Concepts

Setting Up Machines and Collecting Diagnostic Information Using Test Settings

Setting Up Test Machines to Run Tests or Collect Data

Compatibility of Test Settings with Visual Studio 2010

Other Resources

Specifying Test Settings for Visual Studio Tests