Setting Up Machines and Collecting Diagnostic Information Using Test Settings
You can use Test settings in Microsoft Test Manager and Visual Studio to collect extra data when you run your tests. For example, you might want to make a video recording as you run your test. There are diagnostic data adapters to:
Collect each UI action step in text format
Record each UI action for playing back
Collect system information
Collect event log data
Collect IntelliTrace data to help isolate non-reproducible bugs
Diagnostic data adapters can also be used to change the behavior of a test machine. For example, with a test setting in Visual Studio, you can emulate various network topology bottlenecks to evaluate the performance of your team’s application.
In this topic:
With Microsoft Test Manager, you configure a test plan to run your tests. A test plan can have two test settings:
You create these test settings using the Properties page of the test plan in Microsoft Test Manager.
You can configure both of these test settings to use a lab environment which can emulate a single machine, or multiple machine roles. The test setting includes separate configuration settings for the types of data to collect for each machine role using diagnostic data adapters.
What else should I know?
In order to run automated tests in Microsoft Test Manager, you must use a lab environment.
- For additional guidance, see Testing for Continuous Delivery with Visual Studio 2012 – Chapter 3: Lab Environments and Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox.
Use the following topics to help you create and configure test settings in Microsoft Test Manager.
|Create test settings for use in Microsoft Test Manager: You can create separate test settings for manual and automated tests in a test plan.||- Collect more diagnostic data|
- How to: Create Test Settings for a Test Plan
|Update test settings in Microsoft Test Manager: You can make corrections to existing test settings by editing them in the Lab Center or from the Properties activity for a test plan.||- How to: Edit an Existing Test Setting for a Test Plan|
|Choose existing test settings for a test plan in Microsoft Test Manager: Test settings are stored in Team Foundation Server and can be selected for use in multiple test plans.||- How to: Choose Test Settings and Environments for a Test Plan|
A lab environment is a collection of virtual or physical machines that you can use to develop and test applications. A lab environment can contain multiple machine roles needed to test multi-tiered applications, such as workstations, web servers, and database servers. You can create and manage lab environments and run manual tests in a lab environment using Microsoft Test Manager, or run automated tests in a lab environment. When you run your tests using a lab environment, the test will collect data, or affect the behavior of the machine for each specific machine role that you configured in your test settings. In addition, you can use a build-deploy-test workflow with your lab environment to automate the process of building, deploying, and running automated tests on your application.
The following illustration shows examples of test settings and environments for a test plan.
The following illustration shows how you define the set of machine roles for your test settings. You can then select a lab environment that has computers or virtual machines that are assigned to each machine role to use when you run your tests. You can select any lab environment that includes at least the set of machine roles that are defined in your test settings. The lab environment may include other machine roles that are not specified in your test settings, as shown in the following illustration.
To run your unit, coded UI, web performance, or load tests by using Visual Studio, you can add, configure and select the test settings to use when you run your tests. To run your tests, collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings. The test controller will have agents that can be used for each role in your test settings.
What else should I know?
In this version of Visual Studio, there are some compatibility issues of test settings with Visual Studio 2010 to be aware of:
In this version of Visual Studio, test settings are not included in your solution when a unit test project or coded a UI test project is added. This is a result of the enhancements in the Unit Test Framework to improve extensibility, resilience, and performance. However, you can open an existing Visual Studio 2010 test project with a .testsettings file and it will continue to run. Additionally, you can add a .testsettings file to your solution if you need to collect specific data from a data and diagnostic adapter or run your test with test controllers; however, this is not recommended unless necessary.
If you do open a Visual Studio 2010 test project with a .testsettings file, or determine that you must add a .testsettings file to your solution, the procedures for selecting the active .testsettings file, and editing the .testsettings file have changed.
When you add a Web Performance and Load test project to your solution, the solution will still include a .testsettings file; however, the producers for selecting the active .testsettings file for Web performance and load tests, and editing the .testsettings file for Web performance and load tests have changed.
Lab environments are used only when you run your tests by using Microsoft Test Manager and not with Visual Studio.
Use the following topics to help you create and configure test settings for use with Visual Studio.
|Create test settings to use in Visual Studio: You can create a test setting for your unit, coded UI, web performance and load tests using the supported diagnostic data adapters.||- Specifying Test Settings for Visual Studio Tests|
|Edit test settings in Visual Studio: You can make corrections to existing test settings by editing them.||- Editing test settings|
|Specify test setting to use in your solution: You solution can contain multiple test settings files. You can select the test setting file to use, depending on your testing needs from among them.||- How do I use a test settings file?|
|Configure your Visual Studio tests to use remote machines: To run your tests, collect data, or affect a test machine remotely, you can install and configure test controller and test agent.||- Setting Up Test Controllers and Test Agents to Manage Tests with Visual Studio|
The following table provides an overview of the various ways that the diagnostic data adapters can be configured for use with local or remote machine roles.
|Diagnostic data adapter that is used in test setting||Manual Tests on local machine||Automated Tests||Manual Tests: Collecting data by using a set of roles and an environment||Notes|
|Action Log: The action log collects a text description of each UI action that is performed during a test. The action logs are saved with the test results for the test. You can view the action log to see what actions were taken.||Yes||No||Yes (See Notes)||- When you collect data on a remote environment, the recording will work only on the local machine.|
- Action logs are collected if you create an action recording when you run a manual test.
|ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy lets you collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.||Yes||Yes||Yes||- Use this only when either the IntelliTrace or Test Impact diagnostic data adapters are selected for a client role.|
|ASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications.||No||Yes (See Notes)||No||- This diagnostic data adapter is supported only when you run load tests from Visual Studio.|
|Code coverage (Visual Studio 2010): You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.||No||Yes (See Notes)||No||- You can use code coverage only when you run an automated test from Visual Studio or mstest.exe, and only from the machine that runs the test. Remote collection is not supported.|
- Collecting code coverage data does not work if you also have the test setting configured to collect IntelliTrace information. Note: This diagnostic data adapter is only applicable to Visual Studio test settings. It is not used for test settings in Microsoft Test Manager. Additionally, this adapter is for compatibility with Visual Studio 2010 test projects. Note: For compatibility, the code coverage applies when automated tests are run from Microsoft Test Manager or on a remote Test agent from Visual Studio using the legacy MSTest runner.
|Event log: You can configure a test setting to include event log collecting, which will be included in the test results.||Yes||Yes||Yes|
|IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that contains this information. An IntelliTrace file has an extension of .iTrace. When a test fails, you can create a bug. The IntelliTrace file that is saved together with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer. This reduces the risk of a bug being non-reproducible.||Yes||Yes||Yes||- If you enable the collection of IntelliTrace data, collection of code coverage data will not work.|
- If you use IntelliTrace for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.
- Only the following versions of IIS are supported: IIS 7.0, IIS 7.5 and IIS 8.0.
For more information, see Using IntelliTrace.
|Network emulation: You can specify that you want to place an artificial network load on your test by using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. Note:||No||Yes (See Notes)||No||You can use the network emulation diagnostic data adapter for a client or server role. You do not have to use the adapter on both these roles that communicate with each other. Note: This diagnostic data adapter is only applicable to Visual Studio test settings. It is not used for test settings in Microsoft Test Manager. Note: Network emulation cannot be used to increase the network connection speed. Warning: If you include the network emulation diagnostic data adapter in the test settings and you intend to use it on your local machine, then you must also bind the network emulation driver to one of your machine’s network adapters. The network emulation driver is required for the network emulation diagnostic data adapter to function. The network emulation driver is installed and bound to your adapter in two ways: |
|System information: A test setting can be set up to include the system information about the machine on which the test is run.||Yes||Yes||Yes|
|Test impact: You can collect information about which methods of your applications code were used when a test case was run. This can be used together with changes to the application code that was made by developers to determine which tests were affected by those development changes.||Yes||Yes||Yes||- If you are collecting test impact data for a Web client role, you must also select the ASP.NET Client Proxy for IntelliTrace and Test Impact diagnostic data adapter.|
- Only the following versions of IIS are supported: IIS 7.0, IIS 7.5 and IIS 8.0.
|Video Recorder: You can create a video recording of your desktop session when you run a test. The video can help other team members isolate application issues that are difficult to reproduce.||Yes||Yes (See Notes)||Yes||- If you enable the test agent software to run as a process instead of a service, you can create a video recording when you run automated tests.|
To do this, see How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.
What else should I know?
- You can create custom diagnostic data adapters to fulfill a specific testing requirement that is not included in the standard adapters.
You can create and use a custom diagnostic data adapter to collect custom data, or perform tasks to impact a machine such as reduce available system memory, reduce available disk space, or allocate and hold other resources during a test.
Using Microsoft Test Manager you can run manual tests, exploratory test sessions, and automated tests from a test plan. When you run any of these tests from your test plan, the diagnostic data collected will be included in the test results.
After configuring your test settings in Visual Studio, you can run your tests and collect diagnostic data to help isolate bottlenecks and bugs in your application.