Export (0) Print
Expand All

Setting Up Machines and Collecting Diagnostic Information Using Test Settings

Test settings use diagnostic data adapters, which specify various types of data to collect when you run manual tests, automated tests, or both. Diagnostic data adapters can also specify how to affect the test machine. For example, a diagnostic data adapter might create an action recording, an action log, a video recording, or collect system information. Additionally, diagnostic data adapters can be used to simulate potential bottlenecks on the test machine or reduce the available system memory. For example, you can emulate a slow network to impose a bottleneck on the system.

Test settings define the following:

  • The type of tests you will run (manual or automated).

  • The set of roles that are required for your application under test.

  • The role to use to run your tests.

  • The diagnostic data adapters to use for each role.

NoteNote

When you create test settings for your tests by using Microsoft Visual Studio 2010, you can run only automated tests. Therefore, you cannot select the type of tests to run. These test settings files that you create for Microsoft Visual Studio 2010 cannot be used with Microsoft Test Manager.

Running Tests by Using Microsoft Test Manager

If you run your tests using Microsoft Test Manager, you can configure a test plan to use a test setting for all its manual tests, and a test setting for all its automated tests if you need it. In addition, you can select an environment that includes the set of roles in your test settings for all the manual tests, and you must select an environment for all the automated tests in your plan. If necessary, you can override these test settings and environments when you run your tests.

NoteNote

If you run automated tests by using Microsoft Test Manager, you must use an environment.

You configure a test plan by using the Properties page of the test plan in Microsoft Test Manager. The following illustration shows examples of test settings and environments for a test plan.

Test Settings Concepts

NoteNote

Manual tests are always run on a local machine by using Test Runner. For more information, see Running Manual Tests Using Test Runner.

Running Tests by Using Microsoft Visual Studio

If you want to run your tests by using Microsoft Visual Studio 2010, you can configure the test settings to use when you run your tests. If you want to run your tests, collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings. The test controller will have agents that can be used for each role in your test settings. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data.

NoteNote

Environments are used only when you run your tests by using Microsoft Test Manager and not with Microsoft Visual Studio 2010.

Additional Information about Environments

You can create physical or virtual environments by using Microsoft Test Manager. A physical environment might use physical computers or virtual machines. A virtual environment uses only virtual machines. An environment consists of a set of roles. A role specifies the purpose of one or more computers in the environment. For example, a specific role could be "Web Server". You can run tests, collect data, or affect a test machine for each specific role in your environment.

You must specify a test controller for your environment. Each computer with a test agent registered to that controller is called a machine and is available to use for your environment. For more information, see Setting Up Test Machines to Run Tests or Collect Data.

The following illustration shows how you define the set of roles for your test settings. You can then select an environment that has machines that are assigned to each role to use when you run your tests. You can select any environment that includes at least the set of roles that are defined in your test settings. The environment may include other roles that are not specified in your test settings, as shown in the following illustration.

Test Settings and Environments

Test Settings Configuration Details for Diagnostic Data Adapters

The following table provides an overview of the various ways that the diagnostic data adapters can be configured for use with local or remote machines.

Diagnostic data adapter that is used in test setting

Manual Tests on local machine

Automated Tests

Manual Tests: Collecting data by using a set of roles and an environment

Notes

Actions: You can create a test setting that collects a text description of each action that is performed during a test. When you configure this adapter, the selections are also used if you create an action recording when you run a manual test. The action logs and action recordings are saved together with the test results for the test. You can play back the action recording later to fast-forward through your test, or you can view the action log to see what actions were taken.

Yes

No

Yes (See Notes)

  • When you collect data on a remote environment, the recording will work only on the local machine.

ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy lets you collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.

Yes

Yes

Yes

  • Use this only when either the IntelliTrace or Test Impact diagnostic data adapters are selected for a client role.

ASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications.

No

Yes (See Notes)

No

  • This diagnostic data adapter is supported only when you run load tests from Microsoft Visual Studio 2010.

Code coverage: You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.

No

Yes (See Notes)

No

  • You can use code coverage only when you run an automated test from Microsoft Visual Studio 2010 or mstest.exe, and only from the machine that runs the test. Remote collection is not supported.

  • Collecting code coverage data does not work if you also have the test setting configured to collect IntelliTrace information.

IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that has an extension of .iTrace that contains this information. When a test fails, you can create a bug. The IntelliTrace file that is saved together with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer. This reduces the risk of a bug being non-reproducible.

For more information, see Debugging with IntelliTrace.

Yes

Yes

Yes

  • If you enable the collection of IntelliTrace data, collection of code coverage data will not work.

  • If you use IntelliTrace for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.

  • Only the following versions of IIS are supported: IIS 6.0, IIS 7.0 and IIS 7.5.

Event log: You can configure a test setting to include event log collecting, which will be included in the test results.

Yes

Yes

Yes

Network emulation: You can specify that you want to place an artificial network load on your test by using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up.

NoteNote
Network emulation cannot be used to increase the network connection speed.

Yes

Yes

Yes

You can use the network emulation diagnostic data adapter for a client or server role. You do not have to use the adapter on both these roles that communicate with each other.

If you include the network emulation diagnostic data adapter in the test settings and you intend to use it on your local machine, then you must also bind the network emulation driver to one of your machine’s network adapters. The network emulation driver is required for the network emulation diagnostic data adapter to function. The network emulation driver is installed and bound to your adapter in two ways:

  • Network emulation driver installed with Microsoft Visual Studio Test Agent 2010: The Microsoft Visual Studio Test Agent 2010 can be used on both remote machines and your local machine. When you install a Microsoft Visual Studio Test Agent, the installation process includes a configuration step that binds the network emulation driver to your network card. For more information, see Installing and Configuring Visual Studio Agents and Test and Build Controllers.

  • Network emulation driver installed with Microsoft Visual Studio Test Professional 2010: When you use network emulation for the first time, you are prompted to bind the network emulation driver to a network card.

TipTip
You can also install the network emulation driver from the command line on your local machine without installing the Visual Studio test agent by using the following command: VSTestConfig NETWORKEMULATION /install
Caution noteCaution
The Network Emulation adapter is ignored by load tests. Instead, load tests use the settings that are specified in the network mix of the load test scenario. For more information, see Specifying Virtual Network Types in a Load Test Scenario and How to: Create a Test Setting for a Distributed Load Test.

System information: A test setting can be set up to include the system information about the machine on which the test is run.

Yes

Yes

Yes

Test impact: You can collect information about which methods of your applications code were used when a test case was run. This can be used together with changes to the application code that was made by developers to determine which tests were affected by those development changes.

Yes

Yes

Yes

  • If you are collecting test impact data for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.

  • Only the following versions of IIS are supported: IIS 6.0, IIS 7.0 and IIS 7.5.

Video Recorder: You can create a video recording of your desktop session when you run a test. The video can help other team members isolate application issues that are difficult to reproduce.

Yes

Yes (See Notes)

Yes

NoteNote

You can also create custom diagnostic data adapters to fulfill a specific requirement that you have. For more information, see Creating a Diagnostic Data Adapter to Collect Custom Data or Affect a Test Machine.

TipTip

Test Attachment Cleaner

The data that some of the diagnostic data adapters capture can use a lot of database space over time. The administrator of the database used for Visual Studio 2010 cannot control what data gets attached as part of test runs. For example, there are no policy settings that can limit the size of the data captured and there is not a retention policy to determine how long to hold this data before initiating a cleanup. You can use the test attachment cleaner tool to:

  • Determine how much database space each set of diagnostic data captures is using.

  • Reclaim the space for runs that are no longer relevant from a business perspective.

For more information and to download the test attachment cleaner tool, see Test Attachment Cleaner for Visual Studio Ultimate 2010 & Test Professional 2010.

Use the following topics to help you create and configure test settings.

Tasks

Associated Topics

Create test settings to use with manual tests: You can create a test setting for a test plan by using the supported diagnostic data adapters for manual tests.

Create test settings to use with automated tests: You can create a test setting for a test plan by using the supported diagnostic data adapters for automated tests.

Update test settings: You can make corrections to existing test settings by editing them in the Lab Center or from the Properties activity for a test plan.

Choose existing test settings for a test plan: Test settings are stored in Team Foundation Server and can be selected for use in multiple test plans.

Collect your own data or affect a test machine by using custom diagnostic data adapters: You can create and use a custom diagnostic data adapter to perform tasks such as reduce available system memory, reduce available disk space, or allocate and hold other resources during a test.

Running Manual Tests Using Test Runner

You can create manual tests by using Microsoft Test Manager and run the tests by using Test Runner to record whether each step passes or fails. You can save the test outcome and any data that is collected when you run the test.

Running Automated Tests

You can run tests directly from Microsoft Visual Studio 2010, from Team Foundation Build, or from the command line. You can use mstest.exe to run your automated tests from the command line, or you can use tcm.exe to import your test methods into test cases. You can then run the test cases for specific configurations from the command line or Microsoft Test Manager and save the results for the appropriate test plan.

Community Additions

ADD
Show:
© 2014 Microsoft