How to: Create a Test Setting for a Distributed Load Test

Article
02/04/2013

This topic applies to:

Visual Studio Ultimate	Visual Studio Premium	Visual Studio Professional	Visual Studio Express

Test settings can be configured for load tests to enable distributing the load test across multiple machines by using test agents and test controllers. The test settings can also be configured to use diagnostic data adapters, which specify various types of data to collect or how to affect the test machines when you run your load tests from Visual Studio 2010 Ultimate. For example, you can use the ASP.NET Profiler diagnostic data adapter to collect the performance breakdown of the code. Additionally, diagnostic data adapters can be used to simulate potential bottlenecks on the test machine or reduce the available system memory.

Test settings for Visual Studio are stored in a file. They define the following:

The set of roles that are required for your application under test.
The role to use to run your tests.
The diagnostic data adapters to use for each role.

When you run your tests, you select the test settings to use as the active test settings depending on what you require for that specific test run. The test settings file is stored as part of your solution. The file has an extension .testsettings.

When you add a test project to a solution, two default test settings are created. They are added automatically to the solution under the Solution Items folder. If either of these test settings provides what you require when you run your tests, then you can use them by choosing the test settings that you want to be active:

Local.testsettings: This runs your tests locally without diagnostic data adapters selected. This is the default test setting in Visual Studio.
Traceandtestimpact.testsettings: This runs your tests locally with the following diagnostic data adapters configured to collect data from all modules and processes:
- IntelliTrace
- test impact
- system information

The test controller will have agents that can be used for each role in your test settings. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data and Managing Test Controllers and Test Agents.

Use the following procedures to create and remove test settings in your solution for load tests that you plan to run from Visual Studio 2010 Ultimate.

Create a Test Setting for a Distributed Load Test

To add a test settings for a distributed load test

In Solution Explorer, right-click Solution Items, point to Add, and then click New Item.

The Add New Item dialog box appears.
In the Installed Templates pane, click Test Settings.
(Optional) In the Name box, change the name of the test settings file.
Click Add.

The new test settings file appears in Solution Explorer, under the Solution Items folder.

Note

The list of test settings that Visual Studio 2010 Ultimate displays is derived from the list of test settings files in the Solution Items folder. For example, test settings files in the Solution Items folder are displayed when you use the Select Active Test Settings option on the Test menu. This means that if you move a test settings file to another location in your solution hierarchy, it can no longer be used as a test setting from within the Visual Studio 2010 Ultimate integrated development environment.
The Test Settings dialog box is displayed. The General page is selected.

You can now edit and save test settings values.

Note

Each test settings that you create is listed as a choice for the Select Active Test Settings and Edit Test Settings options on the Test menu.
Under Name, type the name for the test settings.
(Optional) Under Description, type a description for the test setting so other team members know what it is intended for.
(Optional) To select the default naming scheme for your test runs, select Default naming scheme. To define your own naming scheme, select User-defined scheme and then type the text that you want in Prefix text. To append the date and time stamp to the test run name, select Append date-time stamp.
Click Roles.

The Roles page is displayed.
To run your tests remotely, or to run your tests remotely and collect data remotely, use the Test execution method drop-down and select Remote execution.
Use the Controller drop-down to select a test controller for the test agents from Controller that will be used to run your tests or collect data.

Note

If this is the first time that you are adding a controller, no controllers will be listed in the drop-down list. The list is populated by previous controllers that you have specified in other test settings. You must type the name of the controller in the box (for example, TestControllerMachine1).

For more information, see Configuring Test Controllers and Test Agents for Load Testing and Using Test Controllers and Test Agents with Load Tests.
To add the roles that you want to use to run tests and collect data, Under Roles, click Add.
Type a name for the role in the Name column. For example, the role might be "Web Server".
Repeat steps 12 and 13 to add all roles that you require.

Each role uses a test agent that is managed by the test controller.
Select the role that you want to run your tests, and then click Set as role to run tests.

Important

The other roles that you create and define will not run tests, but will be used only to collect data according to the data and diagnostic adapters that you specify for the roles in the Data and Diagnostic page.
To limit the agents that can be used for a role, select the role and then click Add in the toolbar under Agent attributes for selected role.

The Agent Selection Rule dialog box is displayed.

Type the name in Attribute Name and the value in Attribute Value, and then click OK. Add as many attributes as you require.

For example, you could add an attribute that is named "RAM > 16GB" that has a value of "True" or "False" to filter on test agent machines that have more than 16GB of memory. To apply the same attribute to one or more test agents, you use the Manage Test Controller dialog box. For more information, see Managing Test Controllers and Test Agents.
Click Data and Diagnostics.

The Data and Diagnostics page is displayed.

In the Data and Diagnostic page, you define what the role does by selecting the diagnostic data adapters that the role will use to collect data. Therefore, if one or more data and diagnostic adapters are enabled for the role, the test controller will select an available test agent machine to collect data for the specified data and diagnostic adapters based on the attributes that you defined for the role. To select the data and diagnostic data adapters that you want to collect for each role, select the role. For each role, select the diagnostic data adapters according to the needs of the tests. To configure each diagnostic data adapter that you have selected for each role, click Configure.

Example of roles and diagnostic data adapters:

For example, you could create a client role that is named "Desktop Client" that has an attribute of "Uses SQL" set to "True" and a server role that is named "SQL Server" that has an attribute set to "RAM > 16GB". If you specify that the "Desktop Client" will run the tests by clicking Set as role to run tests in the Roles page, the test controller will select machines that have test agents that include the attribute of "Uses SQL" set to "True" on which to run the tests. The test controller will also select SQL server machines that have test agents that include the attribute "RAM > 16GB" only to collect data that is defined by the data and diagnostic adapters that are included in the role. The "Desktop Client" tests agent can also collect data for the machines on which it is run if you select data and diagnostic adapters for that role, too.

For details about each diagnostic data adapter and how to configure it, you can view the associated topic in the following table.

For more information about diagnostic data adapters, see Setting Up Machines and Collecting Diagnostic Information Using Test Settings.

Diagnostic Data Adapters for Load Tests

Diagnostic data adapter	Using in load tests	Associated topic
ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy lets you collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.	Unless you have a specific need to collect system information for the test agent machines, do not include this adapter. Caution We do not recommend the use of the IntelliTrace adapter in load tests because of problems that occur because of the large amount of data that is collected. Test impact data is not collected by using load tests.
IntelliTrace: You can configure specific diagnostic trace information that is stored in a log file. A log file has an extension of .tdlog. When you run your test and a test step fails, you can create a bug. The log file that contains the diagnostic trace is automatically attached to this bug. The data that is collected in the log file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this log file the local session can be recreated on another computer. This reduces the risk that a bug cannot be reproduced. For more information, see Debugging with IntelliTrace.	We do not recommend the use of the IntelliTrace adapter in load tests because of problems that occur because of the large amount of data that is collected and logged. You should attempt to use the IntelliTrace adapter only in load tests that do not run long and do not use many test agents.	How to: Collect IntelliTrace Data to Help Debug Difficult Issues
ASP.NET Profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications.	The ASP.NET profiler diagnostic data adapter profiles the Internet Information Server (IIS) process, so it will not work against a development Web server. To profile the Web site in your load test, you have to install a test agent on the machine that the IIS is running on. The test agent will not be generating load, but it will be a collection only agent. For more information, see Installing and Configuring Visual Studio Agents and Test and Build Controllers.	How to: Configure ASP.NET Profiler for Load Tests Using Test Settings How to: Run a Load Test Containing Web Performance Tests that Collects ASP.NET Profiler Data
Code Coverage: You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests.	Although you can use the code coverage adapter in load tests, it will collect code coverage data only on test execution roles and not on server roles that are specified in the test setting.	How to: Configure Code Coverage Using Test Settings for Automated Tests
Event log: You can configure a test setting to include event log collecting, which will be included in the test results.		How to: Configure Event Log Collection Using Test Settings
Network Emulation: You can specify that you want to put an artificial network load on your test by using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. Note Network emulation cannot be used to increase the network connection speed.	The Network Emulation adapter is ignored by load tests. Instead, load tests use the settings that are specified in the network mix of the load test scenario. For more information, see Specifying Virtual Network Types in a Load Test Scenario.
System Information: A test setting can be set up to include the system information about the machines on which the System Information diagnostic and data collector is run. The system information is specified in the test results by using a test setting.	You can collect system information from both the load agents and the system under test.	No configuration is required to collect this information.
Test Impact: You can collect information about which methods of your applications code were used when a test case was run. This can be used together with changes to the application code that are made by developers to determine which tests were affected by those development changes.	Test impact data is not collected with load tests.
Video Recorder: You can create a video recording of your desktop session when you run an automated test. This can be useful to view the user actions for a coded UI test. The video can help other team members isolate application issues that are difficult to reproduce. Note When running tests remotely the video recorder will not work unless the agent is running in interactive process mode.	Caution We do not recommend the use of the Video Recorder adapter for load tests.	How to: Record a Video of Your Desktop as You Run Tests Using Test Settings

Tip

Test Attachment Cleaner

The data that some of the diagnostic data adapters capture can use a lot of database space over time. The administrator of the database used for Visual Studio 2010 cannot control what data gets attached as part of test runs. For example, there are no policy settings that can limit the size of the data captured and there is not a retention policy to determine how long to hold this data before initiating a cleanup. You can use the test attachment cleaner tool to:

Determine how much database space each set of diagnostic data captures is using.
Reclaim the space for runs that are no longer relevant from a business perspective.

For more information and to download the test attachment cleaner tool, see Test Attachment Cleaner for Visual Studio Ultimate 2010 & Test Professional 2010.

Click Deployment.

The Deployment page is displayed.
To create a separate directory for deployment every time that you run your tests, select Enable deployment.

Note

If you do this, you can continue to build your application when you run your tests.
To add a file to the directory you are using to run your tests, click Add file, and then select the file that you want to add.

Note

When you run a load tests, plug-in assemblies, data files, and uploaded files are automatically deployed.
To add a directory to the directory that you are using to run your tests, click Add directory and then select the directory that you want to add.

Note

For more information about how to deploy files and directories for individual tests by using properties and the DeploymentItem attribute, see How to: Configure Test Deployment.
To run scripts before and after your tests, click Setup and Cleanup Scripts.

The Setup and Cleanup Scripts page is displayed.
1. Type the location of the script file in Setup script or click the ellipsis (…) to locate the setup script.
2. Type the location of the script file in Cleanup script or click the ellipsis (…) to locate the cleanup script.
To run your tests by using a different host, click Hosts.
1. In Host Type, verify that the Default is selected.
  
  Note
  
  The ASP.NET in Host type is not supported in load tests.
2. Use the Run test in 32-bit or 64-bit process drop-down to select whether you want the Web performance and unit tests in your load test to run as 32-bit or 64-bit processes.
  
  Note
  
  For maximum flexibility, you should compile your test projects by using the Any CPU configuration. Then you can run on both 32-bit and 64-bit agents. Compiling test projects by using the 64-bit configuration offers no advantage.
(Optional) To limit the time for each test run and individual tests, click Test Timeouts.
1. To abort a test run when a time limit is exceeded, select Abort a test run if the total time exceeds and then type a value for this limit.
2. To fail an individual test when a time limit is exceeded, select Mark an individual test as failed if its execution time exceeds, and type a value for this limit.
Skip Unit Test. Load tests do not use these settings.
Skip Web Test. Load tests do not use these settings.
To save the test settings, click Save As. Type the name of the file that you want in Object name.

Note

If you must change your test settings, click Test and then click Edit Test Settings and point to the test settings that you created. For more information, see How to: Edit an Existing Test Setting for a Test Plan.

To remove a test settings from your solution

Under the Solution Items folder in Solution Explorer, right-click the test settings that you want to remove, and then click Remove.

The test settings file is removed from your solution. This change is reflected in the list of choices for the Select Active Test Settings and Edit Test Settings options on the Test menu.

How to: Create a Test Setting for a Distributed Load Test

Create a Test Setting for a Distributed Load Test

To add a test settings for a distributed load test

To remove a test settings from your solution

See Also

Tasks

Concepts

Other Resources

Additional resources