Setting Up Machines and Collecting Diagnostic Information Using Test Settings

  • Article

You can use Test settings in Microsoft Test Manager and Visual Studio to collect extra data when you run your tests. For example, you might want to make a video recording as you run your test. There are diagnostic data adapters to:

  • Collect each UI action step in text format

  • Record each UI action for playing back

  • Collect system information

  • Collect event log data

  • Collect IntelliTrace data to help isolate non-reproducible bugs

Test settings

Diagnostic data adapters can also be used to change the behavior of a test machine. For example, with a test setting in Visual Studio, you can emulate various network topology bottlenecks to evaluate the performance of your team’s application.

In this topic:

  • Using test settings with Microsoft Test Manager

  • Using test settings with Visual Studio

  • Diagnostic Data Adapter details

Using test settings with Microsoft Test Manager

With Microsoft Test Manager, you configure a test plan to run your tests. A test plan can have two test settings:

  • Manual runs

  • Automated runs

You create these test settings using the Properties page of the test plan in Microsoft Test Manager.

Microsoft Test Manager test settings in test plan

You can configure both of these test settings to use a lab environment which can emulate a single machine, or multiple machine roles. The test setting includes separate configuration settings for the types of data to collect for each machine role using diagnostic data adapters.

Test settings machine roles and adapters

Key tasks for test settings with Microsoft Test Manager

Use the following topics to help you create and configure test settings in Microsoft Test Manager.

Tasks

Associated Topics

Create test settings for use in Microsoft Test Manager: You can create separate test settings for manual and automated tests in a test plan.

Update test settings in Microsoft Test Manager: You can make corrections to existing test settings by editing them in the Lab Center or from the Properties activity for a test plan.

Choose existing test settings for a test plan in Microsoft Test Manager: Test settings are stored in Team Foundation Server and can be selected for use in multiple test plans.

Lab environments

A lab environment is a collection of virtual and physical machines that you can use to develop and test applications. A lab environment can contain multiple machine roles needed to test multi-tiered applications, such as workstations, web servers, and database servers. You can create and manage lab environments and run tests in a lab environment using Microsoft Test Manager. When you run your tests using a lab environment, the test will collect data, or affect the behavior of the machine for each specific machine role that you configured in your test settings. In addition, you can use a build-deploy-test workflow with your lab environment to automate the process of building, deploying, and running automated tests on your application.

The following illustration shows examples of test settings and environments for a test plan.

Test Settings Concepts

The following illustration shows how you define the set of machine roles for your test settings. You can then select a lab environment that has computers or virtual machines that are assigned to each machine role to use when you run your tests. You can select any lab environment that includes at least the set of machine roles that are defined in your test settings. The lab environment may include other machine roles that are not specified in your test settings, as shown in the following illustration.

Test Settings and Environments

Using test settings with Visual Studio

To run your unit, coded UI, web performance, or load tests by using Visual Studio, you can add, configure and select the test settings to use when you run your tests. To run your tests, collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings. The test controller will have agents that can be used for each role in your test settings.

  • What else should I know?
    Notes

    • Prerequsite In Visual Studio 2012, there are some compatibility issues of test settings with Visual Studio 2010 to be aware of:

      • In Visual Studio 2012, test settings are not included in your solution when a unit test project or coded a UI test project is added. This is a result of the enhancements in the Unit Test Framework to improve extensibility, resilience, and performance. However, you can open an existing Visual Studio 2010 test project with a .testsettings file and it will continue to run. Additionally, you can add a .testsettings file to your solution if you need to collect specific data from a data and diagnostic adapter or run your test with test controllers; however, this is not recommended unless necessary.

      • If you do open a Visual Studio 2010 test project with a .testsettings file, or determine that you must add a .testsettings file to your solution, the procedures for selecting the active .testsettings file, and editing the .testsettings file have changed.

      • When you add a Web Performance and Load test project to your solution, the solution will still include a .testsettings file; however, the producers for selecting the active .testsettings file for Web performance and load tests, and editing the .testsettings file for Web performance and load tests have changed.

    • PrerequsiteLab environments are used only when you run your tests by using Microsoft Test Manager and not with Visual Studio.

Key tasks for test settings with Visual Studio

Use the following topics to help you create and configure test settings for use with Visual Studio.

Tasks

Associated Topics

Create test settings to use in Visual Studio: You can create a test setting for your unit, coded UI, web performance and load tests using the supported diagnostic data adapters.

Edit test settings in Visual Studio: You can make corrections to existing test settings by editing them.

Specify test setting to use in your solution: You solution can contain multiple test settings files. You can select the test setting file to use, depending on your testing needs from among them.

Configure your Visual Studio tests to use remote machines: To run your tests, collect data, or affect a test machine remotely, you can install and configure test controller and test agent.

Diagnostic Data Adapter details

The following table provides an overview of the various ways that the diagnostic data adapters can be configured for use with local or remote machine roles.

Diagnostic data adapter that is used in test setting

Manual Tests on local machine

Automated Tests

Manual Tests: Collecting data by using a set of roles and an environment

Notes

Microsoft Test ManagerAction Log: The action log collects a text description of each UI action that is performed during a test. The action logs are saved with the test results for the test. You can view the action log to see what actions were taken.

Yes

No

Yes (See Notes)

  • When you collect data on a remote environment, the recording will work only on the local machine.

  • Action logs are collected if you create an action recording when you run a manual test.

Visual Studio 2012 and Microsoft Test ManagerASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy lets you collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.

Yes

Yes

Yes

  • Use this only when either the IntelliTrace or Test Impact diagnostic data adapters are selected for a client role.

Visual Studio 2012 and Microsoft Test ManagerASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications.

No

Yes (See Notes)

No

  • This diagnostic data adapter is supported only when you run load tests from Visual Studio.

Visual Studio 2012Code coverage (Visual Studio 2010): You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.

No

Yes (See Notes)

No

  • You can use code coverage only when you run an automated test from Visual Studio or mstest.exe, and only from the machine that runs the test. Remote collection is not supported.

  • Collecting code coverage data does not work if you also have the test setting configured to collect IntelliTrace information.

Note

This diagnostic data adapter is only applicable to Visual Studio test settings. It is not used for test settings in Microsoft Test Manager. Additionally, this adapter is for compatibility with Visual Studio 2010 test projects.

Note

For compatibility, the code coverage applies when automated tests are run from Microsoft Test Manager or on a remote Test agent from Visual Studio using the legacy MSTest runner.

Visual Studio 2012 and Microsoft Test ManagerEvent log: You can configure a test setting to include event log collecting, which will be included in the test results.

Yes

Yes

Yes

Visual Studio 2012 and Microsoft Test ManagerIntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that contains this information. An IntelliTrace file has an extension of .iTrace. When a test fails, you can create a bug. The IntelliTrace file that is saved together with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer. This reduces the risk of a bug being non-reproducible.

Yes

Yes

Yes

  • If you enable the collection of IntelliTrace data, collection of code coverage data will not work.

  • If you use IntelliTrace for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.

  • Only the following versions of IIS are supported: IIS 7.0, IIS 7.5 and IIS 8.0.

For more information, see Debug Your App by Recording Code Execution with IntelliTrace.

Visual Studio 2012Network emulation: You can specify that you want to place an artificial network load on your test by using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up.

Note

No

Yes (See Notes)

No

You can use the network emulation diagnostic data adapter for a client or server role. You do not have to use the adapter on both these roles that communicate with each other.

Note

This diagnostic data adapter is only applicable to Visual Studio test settings. It is not used for test settings in Microsoft Test Manager.

Note

Network emulation cannot be used to increase the network connection speed.

Warning

If you include the network emulation diagnostic data adapter in the test settings and you intend to use it on your local machine, then you must also bind the network emulation driver to one of your machine’s network adapters. The network emulation driver is required for the network emulation diagnostic data adapter to function. The network emulation driver is installed and bound to your adapter in two ways:

  • Network emulation driver installed with Microsoft Visual Studio Test Agent 2012: The Microsoft Visual Studio Test Agent can be used on both remote machines and your local machine. When you install a Visual Studio Test Agent, the installation process includes a configuration step that binds the network emulation driver to your network card. For more information, seeInstalling and Configuring Test Agents and Test Controllers.

  • Network emulation driver installed with Microsoft Visual Studio Test Professional 2012: When you use network emulation for the first time, you are prompted to bind the network emulation driver to a network card.

You can also install the network emulation driver from the command line on your local machine without installing the Visual Studio test agent by using the following command: VSTestConfig NETWORKEMULATION /install

Warning

The Network Emulation adapter is ignored by load tests. Instead, load tests use the settings that are specified in the network mix of the load test scenario. For more information, see Specifying Virtual Network Types in a Load Test Scenario and How to: Create a Test Setting for a Distributed Load Test.

Visual Studio 2012 and Microsoft Test ManagerSystem information: A test setting can be set up to include the system information about the machine on which the test is run.

Yes

Yes

Yes

Visual Studio 2012 and Microsoft Test ManagerTest impact: You can collect information about which methods of your applications code were used when a test case was run. This can be used together with changes to the application code that was made by developers to determine which tests were affected by those development changes.

Yes

Yes

Yes

  • If you are collecting test impact data for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.

  • Only the following versions of IIS are supported: IIS 7.0, IIS 7.5 and IIS 8.0.

Visual Studio 2012 and Microsoft Test ManagerVideo Recorder: You can create a video recording of your desktop session when you run a test. The video can help other team members isolate application issues that are difficult to reproduce.

Yes

Yes (See Notes)

Yes

  • If you enable the test agent software to run as a process instead of a service, you can create a video recording when you run automated tests.

To do this, see How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.

  • What else should I know?
    Notes

    • Caution iconWarning There are some compatibility of test settings with Visual Studio 2010 that you should be aware of when working with Visual Studio 2010 test projects in Visual Studio 2012.

    • Caution iconWarning Visual Studio 2012 includes partial support for ARM architecture. The user experience for both manual testing and the new exploratory testing on a machine that has ARM-based architecture is identical to that on other platforms. However, when using a test machine with ARM architecture, you cannot include action recordings. Only the following diagnostic data adapters are supported:

    • Prerequsite You can create custom diagnostic data adapters to fulfill a specific testing requirement that is not included in the standard adapters.

Creating a Diagnostic Data Adapter to Collect Custom Data or Affect a Test Machine

You can create and use a custom diagnostic data adapter to collect custom data, or perform tasks to impact a machine such as reduce available system memory, reduce available disk space, or allocate and hold other resources during a test.

Running Manual Tests Using Test Runner

Using Microsoft Test Manager you can run manual tests, exploratory test sessions, and automated tests from a test plan. When you run any of these tests from your test plan, the diagnostic data collected will be included in the test results.

Running Tests in Lab Environments

After configuring your test setting in Microsoft Test Manager to include a lab environment, you can run either manual or automated tests within the lab environment.

Running System Tests Using Microsoft Visual Studio

After configuring your test settings in Visual Studio, you can run your tests and collect diagnostic data to help isolate bottlenecks and bugs in your application.

External resources

Guidance

Testing for Continuous Delivery with Visual Studio 2012 – Chapter 3: Lab Environments

Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox

See Also

Tasks

How to: Create a Test Setting for a Distributed Load Test

Concepts

Setting Up Test Machines to Run Tests or Collect Data

Compatibility of Test Settings with Visual Studio 2010