Was this page helpful?
Your feedback about this content is important. Let us know what you think.
Additional feedback?
1500 characters remaining
Export (0) Print
Expand All

Specifying Test Settings in Microsoft Test Manager

When you run tests in Microsoft Test Manager, the test framework can collect data such as an action log, a screen and voice recording, or diagnostic trace information for each machine role. You can specify these options in test settings. Test settings control the diagnostic data adapters that actually collect the data.

To test a typical web or distributed system, you will use more than one test machine to perform the roles of client, server, database, and so on. Your test settings specify the roles that are required for your tests, and specify separate diagnostic adapter configurations for each role. When you run your tests in your test plan, a lab environment with the same number of roles is automatically selected. If there are multiple test environments that match the set of roles in your test settings, you can select a different matching environment.

Test settings machine roles and adapters

Requirements

  • Visual Studio Ultimate or Visual Studio Premium or Visual Studio Test Professional

When you are first starting to use Microsoft Test Manager, it is easiest not to modify the default test settings.

If you do not specify test settings, the default one will be used, which will capture system information and an action log, which helps in manual or exploratory testing.

You will have to modify or create a test settings if you want to:

  • Add event logs, IntelliTrace or video recording to your test results and bug work items, to help isolate bugs in your application.

  • Perform test impact analysis to find out what tests are affected by recent code changes.

  • Emulate potential bottlenecks that your app might occasionally encounter in a production environment.

  • Configure details of how automated tests cases are run.

In Microsoft Test Manager, in the Properties page of your Test Plan, you can select two test settings files. One is for manual tests and the other is for automated tests. When you run test cases in that plan, these settings are the default choices, but you can override them in individual runs.

You can either create a new test settings file, or select an existing file that has already been defined in your team project. For example if you are creating a new test plan for the next iteration of your team project, you would typically reuse a test settings file that was used in the current iteration.

To select or create a test settings file for your test plan:

Open Testing Center, Plan, Properties. Under Manual Runs or Automated Runs, click the menu at Test Settings and choose either an existing test settings file, or New. To edit the details of an existing test settings file, you can choose Open.

The settings for Automated Runs are used when you run test cases that have been linked to test code. If all your tests are manual, you don’t have to set this option.

Microsoft Test Manager test settings in test plan

To manage the test settings files of your team project:

Open Lab Center, Test Settings. From there you can edit existing test settings files and create new ones.

Editing an existing test settings

For additional guidance, see Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox.

The test settings pages are:

Provide a name for this settings file, and specify whether it is for manual or automated test runs.

Choose Manual to define settings for exploratory tests, test cases that you run manually by following test steps in Microsoft Test Manager, and tests in which you play back a recorded sequence of actions.

Choose Automated to define settings to run test cases that have been associated with test methods in Visual Studio.

On the Roles page of the test settings, choose a combination of machine roles that is suitable to run your tests. When this test setting is used to run a test, the software under test must be deployed on an environment that has at least the same number of machines, with a matching set of roles. A role is a label such as web server or database server that indicates the intended use of the machine.

The list of available sets of roles is based on the lab environments that are defined in your test project. If you cannot see one that is suitable for your tests, then you must create a new lab environment. For more information, see Creating Lab Environments.

Roles for manual test runs

The Local role is always included. This role corresponds to the computer on which you run Microsoft Test Manager and on which you perform the tests.

You don’t need other roles unless you are testing a distributed or web application and you want to collect diagnostic data from the server machines while you run your tests.

Tip Tip

If you have an environment that includes a desktop client, you can run your manual tests on this machine if you install Microsoft Test Manager. Effectively, this machine then becomes the local machine for your test settings because you will run your manual tests on this machine.

Test settings Roles page

Roles for automated test runs

Automated tests must be run on a lab environment. You must choose a set of roles that includes a role for each machine on which your application is deployed. If it is a simple desktop application, it will require only one role. If it is a distributed application such as a web service, it will also require a role for the web server, and perhaps also for a database server and so forth.

If your application uses an external service that is not part of your application, you should not include that in the set of roles.

At Select the role to use to run your automated tests, choose the machine on which the test code will be loaded and run.

Test setting Roles page

On this page you can add and configure diagnostic adapters to collect data for each machine role in your lab environment. In most cases the diagnostic data is included with the test results.

Select each role in turn and check the diagnostic adapters you want to use.

Test setting Data and Diagnostic page

Diagnostic data adapter

Configuration

Action Log: Allows you to record the actions you perform during your test, so that you can play them back rapidly on a subsequent occasion. The actions are also recorded as text descriptions in any bug report that you create, so that the fault can be more easily diagnosed.

How to: Choose the applications that are recorded in a manual test

ASP.NET Client Proxy for IntelliTrace and Test Impact

Select this adapter in a web client role. It is required if you are testing an ASP.NET application, and you want to collect Test Impact or Intellisense data on the web server role.

Finding theTests that are Affected by Code Changes

How to: Collect IntelliTrace Data to Help Debug Difficult Issues

Event log

The Application, Security or System event logs will be included in the test results. You can write code in your application to add items to these logs.

Choose Configure to select the types of events you want.

WriteEntry

IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that contains this information. The file has the extension of .iTrace. When a test fails, you can create a bug. The IntelliTrace file that is saved together with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file, the local session can be simulated on another computer, which reduces the risk of a bug being non-reproducible.

For more information, see Debug Your App by Recording Code Execution with IntelliTrace.

How to: Collect IntelliTrace Data to Help Debug Difficult Issues

System information: Records information about the machine.

No additional configuration.

Test impact: Enable this option to determine which tests were affected by changes to the code made during development.

For an ASP.NET application, enable this adapter in the web server role, and in the web client role, enable ASP.NET Client Proxy for IntelliTrace and Test Impact.

If you are testing an ASP.NET application, on the role where the IIS server will run, choose Configure, Advanced, ASP.NET.

How to: Collect Data to Check Which Tests Should be Run After Code Changes

Screen and Voice Recorder: You can create a recording of your desktop session when you run a test. The recording can help other team members to isolate application issues that are difficult to reproduce.

To include voice recordings, or specify that you want to save recordings if a test passes in addition to failing, choose Configure. Use Configure to modify the screen recording quality too.

How to: Include Recordings of the Screen and Voice During Tests Using Test Settings

Tip For compatibility information about test settings between Visual Studio 2012 and Visual Studio 2010, see Compatibility of Test Settings with Visual Studio 2010.

These test settings are available only if you are creating a test setting for automated tests. For more information see Creating Automated Tests Using Microsoft Test Manager.

Setting

Purpose

Notes

Deployment

Specify files to be copied to the test machine before running the tests. You can also specify directories to create.

For individual test methods, you can also specify the DeploymentItem attribute in the test code.

For more information, see How to: Deploy Files for Tests.

Scripts

Scripts to be run on the test machine before and after starting automated tests.

Hosts

Configure ASP.NET tests for IIS.

For maximum flexibility, you should compile your test projects with the Any CPU configuration. Then you can run on both 32- and 64-bit agents. There is no advantage to compiling test projects with the 64-bit configuration.

Timeouts

Limit the time the automated tests will run.

Add-ins

Additional configuration for unit tests and web tests.

See Configuring the Unit Test Add-in

If you automate a test case by linking it to a unit test, configure Unit Test on the Add-ins page of the test settings.

  1. For Root folder for the assemblies to be loaded, choose Browse to locate the folder and populate the text box.

    The root folder that is specified can contain environment variables and represents the directory which will be used as the ApplicationBase of the AppDomain that the tests are run in. All the assemblies in this directory will be loadable by your unit tests. In a production environment, a good practice is to set this to the directory where your code under test assemblies are installed. In a development environment, a good practice is to set this to the directory where your code under test assemblies are built to. This makes sure that any references that you have to the product binaries can be loaded and resolved during the discovery and execution of the tests without the need to copy the product binaries around with the tests.

    If no value is set here the ApplicationBase of the AppDomain that the tests are run in is set to the directory which contains the tests.

  2. Select or clear the check box for Use the Load Context for assemblies in the test directory.

    By default, most assemblies are loaded into the correct "Load Context" Usually, you should leave Use the Load Context for assemblies in the test directory selected. However, there are some conditions when you might want to turn this off. If there are a large number of assemblies in your test directory, and you have specified a location under Root folder for the assemblies to be loaded, and your tests are not dependent on being loaded in the Load Context, you could see a performance increase if you do not use the Load Context to load these test assemblies. If your tests depend on being loaded in a context other than the Load Context (not typical).

    For more information, see Best Practices for Assembly Loading.

  3. Under Folders to use when the tests are run, choose Add folder.

    The Browse For Folder dialog box is displayed.

  4. Locate the folder to use and choose OK.

    The Folders to use when the tests are run setting is the setting that you will probably use the most frequently. You can specify multiple paths to folders that assemblies should be resolved from during discovery and execution of the tests. Each of the paths that are specified in this section can contain environment variables. Together with each of the paths that are specified here, there are two options that are associated with it:

    First option   Select the Use Load Context check box to specify that the directory should use the load context when resolving assemblies from the directory (if the load context is not required for the tests to run correctly you might see a performance improvement by clearing this check box).

    Second option   Select the Include sub-folders check box to specify using any sub-folder to include when resolving assemblies from the directory.

  5. Under Additional folders to use when discovering tests, choose Add Folder.

    The Browse For Folder dialog box is displayed.

  6. Locate the folder to use and choose OK.

    This Additional folders to use when discovering tests is useful when you are either executing the tests remotely under Team Build or doing an automated run from Microsoft Test Manager. The paths provided here will be used for assembly resolution, but only during test discovery. These paths can contain environment variables. When tests are being scheduled to execute remotely from a build drop and not all the dependencies of the test assembly are in the same directory, these paths can be used to make sure that MSTest or the Test Controller can find enough of the dependent assemblies to discover the tests and schedule them to the remote machines for execution.

    For runs being scheduled from Microsoft Test Manager, there is an additional token "%BuildDrop%" that can be used to generically refer to the build drop location. This eliminates the need to create or update a Test Settings each time a new build is being tested. Unfortunately this token is not directly supported through Team Build. However, if the build drop location is set in an environment variable named BuildDrop from the build definition, it will have the same result).

  7. Choose Save.

  8. Choose Close.

Show:
© 2015 Microsoft