Export (0) Print
Expand All

Create and run a load test

To determine how well your software responds to various levels of usage, use load tests. A load test models the expected usage of a software program by simulating multiple users who access the program at the same time.

Include your test types in your load test

We’re going to use a sample ASP.NET app. It has three aspx pages – the default page has a radio control to choose red or blue and a submit button. The other two aspx pages are very simple, one has a label named Red on one and Blue on the other. When you choose submit on the default page, we display one of the other pages. Create an app like that, download our sample, or just follow along with your own web app.

Running the web app to be tested

Your solution should also include a web performance test that browses through the pages of the web application similar to the ColorWebAppTest project created in Record and run a web performance test.

Solution with web performance test

  1. If you don’t have Visual Studio Ultimate, get it here.

  2. Add a load test to your web performance and load test project.

    Add a load test to the test project

  3. Name the scenario and configure think times.

    Think times represent the time that a user would ponder a web page before going on to the next page.

    Name the scenario and configure think times

  4. Choose and configure the load pattern type.

    The load pattern properties specify how the simulated user load is adjusted during a load test.

    Configure the load pattern

  5. Configure the test mix model.

    The test mix model specifies the probability of a virtual user running a test in the test mix of a load test scenario.

    Configure the test mix model

  6. Add tests to the test mix.

    Add tests to the test mix

    In this case, we’ll add the web performance test ColorWebAppTest.

    Add tests to test mix

    If the load test contained more tests, you could use the sliders to adjust the test distribution.

    Test mix distribution

  7. Add and distribute network types.

    Configure the network mix
  8. Add and distribute the browser types.

    Add and distribute browser types

  9. The counter sets step allows you to monitor performance counters on test controller and agent machines in a distributed load test.

    For this example, we’ll leave the defaults and skip this step; however, if you want to learn more see Q: Can I increase the capacity of my load tests?.

    Add computers and counters to monitor

  10. Select run duration to 2 minutes to run a smoke test.

    When you build your tests, it is a good practice to validate that everything is configured correctly and running as expected by running a short, light test. This process is known as smoke testing.

    Set the run duration

    Run settings are a set of properties that affect an entire load test. The run settings determine the length of the test, warm-up duration, maximum number of error details reported, sampling rate, description, whether to save the log on a load test failure, and the validation level.

  11. Choose Finish to create the load test.

  12. Run the test.

    Run the load test

    Analyze the test while it runs.

    Analyze the test while it is running

    After the test completes, a detailed summary report is displayed.

    Load test summary report displays

  13. View the activity details for each of the individual virtual users.

    Detail view shows activity for each virtual user

Let’s add a threshold violation to demonstrate how load tests can help isolate performance issues in your application.

  1. In Solution Explorer, expand the ColorWebApp ASP.NET Web application project folder and then expand the Red.aspx node.

  2. Right-click the Red.aspx.cs file and select View Code.

  3. In the Code Editor, add the following highlighted code in the Page_Load method:

    protected void Page_Load(object sender, EventArgs e)
            {
                Random rnd = new Random(); 
                int result = rnd.Next(5000); 
                    //To emmulate various page times up to 5 seconds. 
                    Thread.Sleep(result);  
                if (result < 500) 
                    Response.Redirect("NOWHERE");  //10% chance to cause an HTTP 404 error.
            }
    

    This additional code is so that you can view mock threshold violations and errors in the Load Test Analyzer. You will view these violations later in this walkthrough.

  1. In the Counter Sets node, expand the LoadTest counter set node and then expand the Counter Categories folder node.

  2. Expand the LoadTestPage counter category node and then expand the Counters folder node.

  3. Right-click the Avg Page Time counter node and select Add Threshold Rule.

  4. The Add Threshold Rule dialog box is displayed.

  5. Under Select a rule, leave the Compare Constant rule selected.

  6. Under Properties for selected rule, in the Options category, set Alert If Over to True.

  7. Under the Threshold Values category, set the Warning Threshold Value to 3 and the Critical Threshold Value to 4.

  8. Choose OK.

  9. The Avg Page Time counter has a Threshold Rules folder added under it with the new rule.

After you have created the Load test, run it to view how your Web site responds to the load simulation. While the load test is running, you can start some initial analysis in the Load Test Analyzer window.

  1. With a Load test open in the Load Test Editor, choose the green Run button. Your load test starts to run in the Load Test Analyzer in the Graphs view.

  2. While the test is running, choose the Show Legend drop-down list button on the toolbar and select Show Threshold Violations On Graph.

    If your test simulation exceeds any thresholds, icons appear in the tree control nodes to indicate a threshold violation. Errors have a red circle overlay and warnings have a yellow triangle overlay.

  3. When you see a threshold violation icon appear on the Graphs view's Page Response Time graph, in the Load Test Analyzer's toolbar, choose Add Analysis Notes.

    The Analysis dialog box is displayed.

  4. In the Description text box, type Threshold violation.

  5. In the Analysis test box, type Suspected code defect in Red.aspx.cs file.

    The comment will be permanently saved with the load test results.

  6. After the load test is finished running, the load test results are presented in a separate tab displaying LoadTest1[time] in the Summary view in the Load Test Analyzer.

  7. From the Web performance and load test project, open a load test.

  8. With a load test open in the Load Test Editor, choose the Run button on the toolbar.

    -or-

    From the LOAD TEST menu, choose Run or Debug and then chose either Selected Test or All Tests in Solution.

    Tip Tip

    You can select one or more load tests in your solution and choose Selected Test.

    For more information, see How to: Run Tests from Microsoft Visual Studio.

  9. You can use the Load Test Analyzer to start analyzing your load test data while it is running.

  10. Use the Graph Options drop-down on the Load Test Analyzer toolbar to switch between collapsing and scrolling modes while the load test is running.

  11. You can add a comment while the load test is running that will be stored permanently with the load test result.

    For more information, see [retired] How to: Add a Comment to a Running Load Test Using the Load Test Analyzer.

    After a load test has completed, The Load Test Analyzer appears as a new tabbed document with the load test summary displayed. The Load Test Analyzer can also be docked or set to float by using the usual Visual Studio window manipulation techniques. The title of the window is the name of your load test and the time that the test was started, for example, LoadTest2 [1:15 PM].

    For more information, see Load Test Analyzer Overview.

    The load test result for the completed load test contains performance counter samples and error information. This information was collected periodically from the computers-under-test. A large number of performance counter samples can be collected over the course of a load test run. The amount of performance data that is collected depends on the length of the test run, the sampling interval, the number of computers under test, the number of counters being collected, the data collectors that are configured, and the logging levels. For a large load test, the amount of performance data that is collected can easily be several gigabytes. For more information, see Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents and Considerations for Load Testing.

  12. While a test is running, a condensed set of the performance counter data that can be monitored in the Load Test Analyzer is maintained in memory. To prevent the resulting memory requirements from growing unbounded, a maximum of 200 samples for each performance counter is maintained. This includes 100 evenly spaced samples that span the run's current elapsed time, and the most recent 100 samples. The result that is accumulated during a run is called an in-progress load test result.

    In addition to the condensed set of performance counter data, the Load Test Analyzer has the following functionality available to analyze the in-progress load test result data that is unique while a load test is running:

    • A progress indicator specifies the time that remains.

    • A button on the Load Test Analyzer toolbar is available to stop the load test.

    • You can specify either collapsing or scrolling graphing modes on the Load Test Analyzer toolbar:

      • Collapsing is the default graph mode in the Load Test Analyzer during a running load test. A collapsing graph is used for load test while it is running to reduce the amount of data that must be maintained in memory, while still showing the trend for a performance counter over the full run duration.

      • Scrolling graph mode is available when you are viewing the result of a load test while it is running. A scrolling graph is an optional view which shows the most recent data points. Use a scrolling graph to view only the most recent 100 data intervals in the test.

    • An Overview pane which displays the configuration, requests and test cases information for the running load test.

After the load test is finished, you can continue further analysis of the load test results. For more information, see Load Test Analyzer Overview.

  1. In the Summary view, scroll down to the table titled Errors and choose either the Http Error or the Validation Rule Error.

  2. The Load Test Analyzer changes to the Tables view with the Errors table displayed and the rule type that you clicked is selected.

  3. In the row for the Validation Rule Error rule type, notice the ValidateResponseurl listed under the SubType error column. This threshold violation was caused by the following highlighted code that you added to the Red.aspx.cs file which randomly causes a mock page delay for up to five seconds:

    protected void Page_Load(object sender, EventArgs e)
            {
                Random rnd = new Random(); 
                int result = rnd.Next(5000); 
                    //To emmulate various page times up to 5 seconds. 
                    Thread.Sleep(result); 
                if (result < 500)
                    Response.Redirect("NOWHERE");  //10% chance to cause an HTTP 404 error.
            }
    

    This mock delay can potentially violate both the Warning Threshold Value of 3 and the Critical Threshold Value of 4 you specified earlier in the walkthrough. The warning icons are displayed as yellow triangles with an exclamation point in them and the critical violation icons are displayed as red circles with an X in them.

  4. In the row for the Http Error error type, notice the 404 - NotFound under the SubType error column. This was caused by the following highlighted code that you added to the Red.aspx.cs file. This code produces a ten per cent chance of redirecting to a nonexistent page, which causes the error:

    protected void Page_Load(object sender, EventArgs e)
            {
                Random rnd = new Random();
                int result = rnd.Next(5000);
                    //To emmulate various page times up to 5 seconds.
                    Thread.Sleep(result); 
                if (result < 500) 
                    Response.Redirect("NOWHERE");  //10% chance to cause an HTTP 404 error.
            }
    
  5. Under the Count column, choose the link the number for the Validation Rule Error rule type.

    The Load Test Errors dialog box is displayed.

  6. Scroll to the right and under the Details column, choose the TestLog link.

  7. The Web Performance Test Viewer opens in a separate tab displaying the ColorWebTest associated with the error.

  8. Choose Close on the Load Test Errors dialog box.

  9. Select the LoadTest[time] tab to go back to the Load Test Analyzer that displays the load test results.

  10. In the Counters panel, observe that the Scenario basic stress node has one of the threshold warning icons on it. Expand the node until you get to the Avg Page Time counter that was impacted by the code that caused the threshold violations.

  11. In the Graphs view, notice that the threshold violation icons are also displayed for the threshold violations appearing in the Page Response Time graph.

  12. Choose the plot line that has the violation icon on it.

  13. The plot line is bolded and the Avg Page Time counter is highlighted in the Graphs view Legend for the Red (Reporting name added in previous walkthrough) request.

  14. Notice that the Max column for the Avg Page Time counter exceeds the threshold you specified.

A: Failures can be caused by the following:

  • Bad tests in the test mix: Verify that all the web performance, unit, and coded UI tests that are contained in the test mix will pass when they are run by themselves. For data-bound web performance tests, run through all of the data values.

  • Unable to use SQL Tracing: When you run a load test locally, with SQL tracing enabled, you might receive the following message:

    Error occurred running test. Could not start SQL tracing: You do not have permission to run 'SP_TRACE_CREATE'

    To use SQL tracing in a load test that is run locally on a computer that is running the Windows Vista operating system, you must be a member of the sysadmin role on the instance of SQL Server that is being traced. To fix this problem, a SQL Server administrator must add you to the sysadmin role.

  • Error occurred running test. (Computer xyz) could not access the result repository: Invalid object name 'LoadTestRun': This error indicates that the load test database schema was not created. You can use Query Analyzer to run the LoadTestResultsRepository.Sql file located in <Visual Studio install folder>\Common7\IDE\ to create the database.

    If you are using SQL Express, you can run sqlcmd -S .\SQLEXPRESS -i loadtestresultsrepository.sql at a command prompt in the directory listed previously.

    Caution note Caution

    The parameters are case sensitive. You must type uppercase S and lowercase i.

    For more information, see How to: Create a Load Test Results Repository Using SQL.

  • LoadTestCounterNotFoundException error: This error occurs when a performance counter that is included in one of the counter sets in your load test is not found in the performance counter category that contains it. If this is a counter that you added to the counter set, the name of the performance counter is possibly misspelled. It is also possible that the performance counter no longer exists in the category because the performance counter has been removed in a newer revision of the software component that defines the performance counter. You can remove it from the counter set to correct the error without losing any useful data.

  • LoadTestResultsCollectorSlowException error: This error indicates that the test controller was not able to collect performance counter results from all the computers in the specified sample rate for the load test. This can occur when there are many performance counters to collect from many different computers as specified by the counter set mappings for the load test. It can also occur when the test agent is running on the same computer as the test controller. You might be able to correct this error by increasing the sample rate for the load test.

  • LoadTestLimitExceededException error: This error occurs whenever 1000 or more errors of the same type occur. It usually indicates that there is a problem with the test that is running under the load test. For example, if your Web performance test issues requests to URLs that are not found, you should correct the Web performance test to fix this error.

  • Unable to access the load test results repository: When you run a load test, you might receive the following message:

    Could not access the load test results repository

    One possible cause of this error is specifying the incorrect case for the parameter names when you use the SQLCMD command line utility to set up your load test results repository. The following code is a sample command to set up a load test results repository on a server named ContosoServer1:

    SQLCMD -S ContosoServer1 -U <user name> -P <password> -i loadtestresultsrepository.sql

    Caution note Caution

    The parameters are case sensitive. You must type uppercase S, U, and P, and lowercase i.

    For more information, see How to: Create a Load Test Results Repository Using SQL.

  • Unable to Generate Expected Load: A common issue when you run a load test is not being able to generate the load you expect. The following table lists some possible causes of this problem:

    Maximum load is being limited by the think time or by the number of virtual users.

    If think time is turned on it can limit the rate at which each virtual user can submit requests. For example, 5 seconds of think time per request yields a maximum of 0.2 requests per second per virtual user. You can try one of the following changes, in order of preference:

    1. Increase the number of virtual users for more realistic load generation. Increasing the number of virtual users usually requires more memory.

    2. Reduce think time.

    3. Turn off think time for maximum load generation.

    Caution note Caution

    Turning off think time can have a large impact on the test engine If you turn off think time, reduce the number of virtual users.

    The proxy property of your Web performance test is set to "default".

    Using "default" as the proxy setting in a Web performance test is convenient because it enables automatic proxy server detection. However, using "default" as the proxy setting can cause performance problems in load tests and will greatly reduce your maximum throughput. It is better not to use a proxy server when you run a load test. If a proxy server is required, specify the name of the proxy server instead of "default".

    Application bottlenecks.

    Remember, the load testing tool is designed to find bottlenecks in your application. If you have pages with high response times because of a database or CPU bottleneck, it will limit the number of requests per second that each virtual user can issue. Start with a small amount of load and make sure response time stays reasonable as you increase the load slowly. You can use the Response Time Goal property to set the maximum expected response time on each request.

    The CPU, memory, or network of the Web server has exceeded its limit.

    If the CPU, memory, or network of the Web server has exceeded its limit, you might not be able to generate the load you expect. It is possible that you have found the load limit of the server. You can increase the CPU, memory, or network of the Web server.

    The CPU, memory, or network of the computer generating the load has exceeded its limit.

    You might need more powerful computers, or more test agent computers, to generate the desired load.

    The CPU, memory, or network of the database server (if applicable) has exceeded its limit.

    If the CPU, memory, or network of the database server has exceeded its limit, you might not be able to generate the load you expect. It is possible that you have found the load limit of the database server. You can increase the CPU, memory, or network of the database server.

  • Load generation limitations on multi-core computers: When you run load tests on multi-core computers, load generation is limited as follows:

    • If the computer is running Visual Studio Ultimate the load generation is limited to one core.

    • If the computer is running Visual Studio Test Agent, the load generation is not limited; it runs on all cores and processors.

A: Choose one of the following load patterns for each scenario in your load test that is appropriate for your test goals:

  • Constant: A constant load pattern is used to run the same user load during the run of a load test. Be careful about using a constant load pattern with a high user count; doing this could place an unreasonable and unrealistic demand on your server or servers at the beginning of the load test. For example, if your load test contains a Web test that starts with a request to a home page, and you set up the load test with a constant load of 1,000 users, the load test will submit the first 1,000 requests to the home page as fast as possible. This might not be a realistic simulation of real-world access to your Web site. To alleviate this, consider using a step load pattern that ramps up gradually to 1,000 users, or specify a warm-up period in the Load Test Run Settings. If a warm-up period is specified, the load test will increase the load gradually during the warm-up period.

  • Step: A step load pattern can be used to increase the load on the server or servers as the load test runs so that you can see how performance varies as the user load increases. For example, to see how your server or servers perform as the user load increases to 2,000 users, you might run a 10-hour load test using a step load pattern with the following properties:

    • Initial User Count: 100

    • Maximum User Count: 2000

    • Step Duration (seconds): 1800

    • Step Ramp Time (seconds): 20

    • Step User Count: 100

    These settings have the load test running for 30 minutes (1800 seconds) at user loads of 100, 200, 300, up to 2,000 users. The Step Ramp Time property is worth special mention here because it is the only one of these properties that is not available in the New Load Test Wizard. This property allows the increase from one step to the next (for example from 100 to 200 users) to be gradual instead of immediate. In the example, the user load would be increased from 100 to 200 users over a 20 second period; this is an increase of 5 users every second.

  • Goal: A goal-based load pattern is useful when you want to determine the number of users that your system can support before reaching some level of resource utilization. This option works best when you have already identified the limiting resource, that is, the bottleneck, in your system. For example, if you know that the limiting resource in your system is the CPU on your database server, and you want to see how many users can be supported when the CPU on the database server is approximately 75% busy, you could use a goal-based load pattern with the goal of keeping the value of the performance counter "%Processor Time" between 70% and 80%.

    If some other resource is limiting the throughput of the system, the goal specified by the goal-based load pattern might never be reached, and the user load will continue to increase until the value specified for the Maximum User Count is reached.

    This is usually not the desired load. Therefore, be careful about the choice of the performance counter in the goal-based load pattern, and also make a conscious decision about the value for the Maximum User Count to place an upper bound on the user load.

A: The browser mix can include the following browser types:

  • Chrome 2

  • Firefox 3.0

  • Internet Explorer 5.5

  • Internet Explorer 6.0

  • Internet Explorer 7.0

  • Internet Explorer 8.0

  • Internet Explorer 9.0

  • Internet Explorer 10.0

  • Netscape 6.0

  • Pocket IE 3.02

  • Safari 3

  • Safari for iPhone

  • Smartphone

A: Tests can include the following network emulation types:

  • LAN (Default, does not apply to this troubleshooting topic)

  • 3G

  • Cable-DSL-1.5Mbps

  • Cable-DSL-768k

  • Cable/DSL-384k

  • CDMA

  • Dial-up 56k

  • Intercontinental slow WAN 300 Kbps

  • Intercontinental WAN 1.5 Mbps

  • Intracontinental WAN 1.5 Mbps

A: Network emulation simulates network conditions by direct manipulation of the network packets. The network emulator can emulate the behavior of both wired and wireless networks by using a reliable physical link, such as an Ethernet. The following network attributes are incorporated into network emulation:

  • Round-trip time across the network (latency)

  • The amount of available bandwidth

  • Queuing behavior

  • Packet loss

  • Reordering of packets

  • Error propagations

Network emulation also provides flexibility to filter network packets that are based on IP addresses or protocols such as TCP, UDP, and ICMP.

Network emulation can be used by network-based developers and testers to emulate a desired test environment, assess performance, predict the effect of change, or make decisions about technology optimization. When compared to hardware test beds, network emulation is a much cheaper and more flexible solution.

How Network Emulation Works in Load Tests

When you start a load test, it allocates a range of available ports for each Network profile that you have selected in your network mix ,for example, DSL and 56.K Modem. This port range is available to the network emulation driver that is enabled at run time (by default, the network emulation driver is disabled).

During load testing, when the load generator sends a request to the application under test, it specifies a port from the port range. When the network emulation driver detects this port from the select port range, it can associate this port with the network profile that this request should follow. This enables the driver to throttle the load in software to make sure that it meets the network profile that you have selected.

A: Use the following information:

Often, one symptom you will see is load test records socket exceptions in the log, such as the following:

"The requested address is not valid in its context xx.xx.xx.xxx:80"

Note Note

Other conditions could cause such socket exceptions also. The load test might continue to work but the socket exceptions are logged. The next section will help you isolate and troubleshoot the problem.

To troubleshoot and isolate problems effectively you must make sure that you have completed the basic tests.

  1. Verify that you have full network connectivity across all machines that are participating in your load test.

  2. Make sure that you have configured the network emulation correctly by following the instructions and verifying that administrator rights are available for the test agent.

  3. Check whether all firewalls are disabled when you are troubleshooting to make sure that a firewall is not blocking specific ports or traffic on the network.

    1. Run TCPView to make sure that any socket connections are actually visible during run time Look for "red" highlights.

      Tip Tip

      You may use other port monitoring tools, for example Portmon.

  4. Make sure that no virus software on the load generator machine is obstructing this software.

  5. To isolate whether the problem is with the Network Emulation Driver or the Load Test Components, follow these steps:

    1. Eliminate the network emulation driver as a cause:

      1. Run the load test with network emulation configured correctly, even though you might be seeing socket exceptions.

      2. Ping another host to see whether the output shows network slowdown, higher latency, or both. Check whether the delay value matches the selected network profile. If the latency values match the profile that you have selected, the network driver is working well.

      3. From that test agent machine where you are running the load test, attempt a connection to any host outside,such as your favorite Web page. This test verifies that, when the load test is running and the network driver is enabled, external or lab connectivity is not a problem. This will eliminate your network emulation driver as a problem area.

  6. Eliminate the Load Test Components as a cause:

    1. You can download and run Sendrequests.exe on the same machine as the load generator (test agent machine). Sendrequests.exe is a sample program to troubleshoot socket exceptions during network emulation load tests.

      Caution note Caution

      The Sendrequests.exe program is not supported by Microsoft.

      This sample program simulates the exact set of socket connection calls that are used in the load testing components. If this test program also displays socket exceptions, this eliminates the load testing product as a cause for the socket exceptions. The socket exceptions also indicate that the problem is occurring in either the environment, machine, network, or something external to the tooling.

      Please debug the external problem first before you try to run the load test again.

    2. If this sample program is working correctly, you will see the output as shown in the following illustration. This will confirm that a problem likely is occurring in the load test program and that the environment is not the likely cause.

      Sendrequests.exe success output

      SendRequests output

IPSEC Not Compatible with Network Emulation

If IPSEC is enabled, the ports in the network packet are encrypted. Therefore the network emulation driver will not be able to determine that the packets are from the designated port range as set by the load test engine that was described previously in How Network Emulation Works in Load Tests. You must disable IPSEC for network emulation to work.

A: Yes, see Create and run a load test.

When you use Team Foundation Build to run a load test that was created by using the default settings, the default counters do not appear automatically in the test results. To view the counters, drag the required counters onto the load test results graph.

A: Yes, using coded UI tests can be useful because coded UI tests let you capture performance at the UI layer. For example, if you have an application that takes 1 second to return data to the client but 8 seconds to render the data in the browser, you cannot capture this type of performance problem by using a web performance test.

Be aware of the following prior to including coded UI tests:

  • The inclusion of coded UI tests should only be done if the coded UI tests are created with the intension of being used as performance tests (not a UI validation test).

  • Coded UI Tests drive the mouse and keyboard. Therefore, only 1 virtual user can run coded UI tests per agent. The best way to control this is to set up a separate scenario in the load test, and set the user load to 1 user. If you have more than one UI test, configure the test mix as Sequential. For more information, see [retired] Creating Additional Scenarios for an Existing Load Test and Editing Text Mix Models to Specify the Probability of a Virtual User Running a Test.

  • You need to configure the load agents to run as an interactive process rather than as a service. For more information, see Installing and Configuring Test Agents and Test Controllers.

  • By default, you will not get accurate timing measurements from a coded UI test that is used in load tests because the calls are asynchronous. You must implement your coded UI tests correctly to obtain accurate timing measurements. This can be accomplished by using the WaitForControlReady method. The following sample code snippets demonstrate this for a login page:

    1. This is a simplified example. A real test would also have to handle timing if the login failed.

      Time how long it takes to load the login page.

                  TestContext.BeginTimer("UI Login Page Load");
      

    2. This call loads the login page.

                 UIMap.LoginPage();
      

    3. Any timing taken in a Web test must use WaitForReady. This will wait until the form is displayed.

                 UIMap.UIHttpteamtestweb1STORWindow1.UIHttpteamtestweb1STORDocument.UIEmailEdit.WaitForControlReady();
                 TestContext.EndTimer("UI Login Page Load");
      

      Caution noteCaution

      Be sure that the time that you spend filling out the form is not included in the timer. While recording, generate code from the recorder after filling out a form, but before submitting it.

    4. This function fills out the login form.

                 UIMap.FilledInLoginForm();
      

    5. Time the login operation

                 TestContext.BeginTimer("UI Login");
                 UIMap.LoggedIn();
      

    6. Any timing taken in a Web test must use WaitForReady. This waits until the login confirmation page is displayed.

                 UIMap.UIHttpteamtestweb1STORWindow.UIHttpteamtestweb1STORDocument.UIWelcomeJamesBondwellPane.WaitForControlReady();
                 TestContext.EndTimer("UI Login");
      

A: Yes, you can use a test controller and agent machines, or you can avoid dealing with managing the machines by using the cloud-based load testing service.

Test controller and agents

After you are confident that your test runs well with a light load and for brief durations, you can run your load test on a test controller and test agents. This group of computers generates a simulated load for testing.

Setting Up Test Machines to Run Tests or Collect Data , Installing and Configuring Test Agents and Test Controllers, and Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents.

Do not overload test agents: If a test agent machine has more than 75% CPU utilization or has less than 10% of physical memory available, add more agents to your load test to ensure that the agent machine does not become the bottleneck in your load test.

For more information, see How to: Specify Test Agents to Use in Load Test Scenarios and Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents.

New load test wizard step

  1. When you create a new load test, setup the machines that you want to collect performance counter data from.

    In the Counter Sets step of the wizard, add the computers and counters to monitor.

  2. Counter sets are gathered on computers that you specify. The association between a counter set and a computer that is used during a load test is a counter set map. For example, the Web server that you are testing might have ASP.NET, IIS, and .NET application counter set mappings.

    You can select the computers to monitor during test runs by choosing Add Computer and typing the name of the server that hosts the non-production Web site that you targeted earlier. By adding the host computer name, you gather performance information that is important in your load test.

    Note Note

    On each server that you monitor, you must have sufficient user permissions to run performance monitors. Otherwise, errors are generated.

    You can add a separate entry for a computer that hosts the SQL database for the site. If you choose not to add any computers, only local load counters are added to your tests.

    You can then select the counter sets that you want to monitor. A set of predefined counter sets that add specific performance monitors to your load test are displayed which includes Application, ASP.NET, .NET Application, IIS and SQL.

To run a load test on a test controller and test agents

  1. Open a Web performance and load test project that contains a load test.

  2. In Solution Explorer, select the test settings under Solution Items, open its shortcut menu, and then choose Open.

    The Test Settings dialog box is displayed. For more information, see Specifying Test Settings for Visual Studio Tests.

  3. Select Roles. From the Test execution method drop-down list, select Remote execution.

  4. From the Controller drop-down list, select the controller. Alternatively, you can type the name of the controller.

  5. Either choose Save As and specify a name for a new test settings.

    -or-

    Choose Apply to save the changes to the current test settings.

  6. On the Test menu, point to Select Active Test Settings and then select the test settings on the submenu that you just saved.

  7. Run your tests from the Load Test Editor or from the LOAD TEST menu. For more information, see [retired] How to: Run a Load Test.

    Your test runs on the test controller and test agents.

A: Yes, use the mstest.exe command line tool.

  1. Open a Visual Studio command prompt.

    Open the developer command prompt

  2. Change directory to the folder that contains the test. For example: CD C:\Users\abarrl\Documents\Visual Studio 2013\Projects\ColorWebApp\TestResults.

  3. Use the mstest.exe using one of the following procedures.

    • Run a single test: Use the /TestContainer argument. A .loadtest file or .webtest file is considered a test container and a DLL that contains unit tests is also a test container. For example, if you have a web performance test called WebTest1.webtest, use the following:

      mstest /TestContainer:LoadTest1.loadtest

    • Run multiple tests: Use the /TestContainer argument multiple time. For example, if you want to execute LoadTest1.loadtest and LoadTest2.loadtest, use the following:

      mstest /TestContainer: LoadTest1.loadtest /TestContainer: LoadTest2.loadtest

    • Run a distributed test using a test controller and test agents: When you run a test, you need to create or use a test setting that has a test controller specified in it by using the /testsettings parameter.

      mstest /TestContainer:LoadTest1.loadtest /TestSettings:NewOrEditedTestSetting.testsettings

    • Specify a test results file: Results file (.trx file) are saved using a unique name that contains user, machine and a timestamp. If you want to specify the name of the results file and where it is generated, use the /resultsfile parameter.

      mstest /TestContainer:LoadTest1.loadtest /resultsfile:c:\results\MyResults.trx

A: Yes. if you have a 64-bit machine, you can run load tests as a 64-bit process by editing the test setting.

  1. If your coded web performance test or unit tests were compiled as 32-bit/x86, recompile them using a 64-bit/x64 configuration.

  2. Open the Test Settings file for your tests and select Run tests in 64 bit process on 64 bit machine.

    Select 64-bit

A: Choose a value for the Sample Rate property in the load test run settings based on the length of your load test. A smaller sample rate, such as the default value of five seconds, requires more space in the load test results database. For longer load tests, increasing the sample rate reduces the amount of data that you collect.

Here are some guidelines for sample rates:

Load Test Duration

Recommended Sample Rate

< 1 Hour

5 seconds

1 - 8 Hours

15 seconds

8 - 24 Hours

30 seconds

> 24 Hours

60 seconds

A: Yes, there is a set of properties in the run settings that allows the SQL tracing feature of Microsoft SQL Server.

The user who is running the load test must have the SQL privileges needed to perform SQL tracing, and a directory where the trace file will be written must be specified.

  1. Right-click the active Run Settings node for your load test and then choose Properties.

  2. Set the SQL Tracing Enabled property. True indicates that SQL Tracing is enabled; False indicates that it is not.

  3. Set the SQL Tracing Connect String property. Type the location of the trace file, or choose the ellipsis button to open the Connection Properties dialog box.

  4. Set the SQL Tracing Directory property. Type a folder in which to store the SQL trace data. The path must be accessible to the SQL Server and the client that is running Visual Studio Ultimate.

  5. Set the Minimum Duration of Traced SQL Operation property. Type a value for the minimum duration of traced queries. For example, 500 indicates that all queries that take longer than 500 are traced. The units are in milliseconds.

    Note Note

    If you are using SQL Server 2005, the units of duration are in microseconds.

  6. Save and run your test.

    You can view the SQL Tracing data only after your load test has completed. For more information, see The SQL Trace Data Table.

A: Yes, by default the Timing Details Storage property in the run setting is enabled. This allows 90th and 95th percentile data to be shown in the Load Test Analyzer’s Tests, Transactions, and Pages tables.

There are two choices for enabling the Timing Details Storage property:

  • StatisticsOnly: As soon as the percentile data has been calculated, the individual timing data is deleted from the repository. This reduces the amount of space that is required in the repository when you use timing details.

  • AllIndividualDetails: The timing detail data is available for that processing. Use this if you want to process the timing detail data in other ways, for example with SQL tools.

    Additionally, you can analyze the virtual user activity using the Virtual User Activity chart in the Load Test Analyzer after the load test completes running. For more information, see Analyzing Load Test Virtual User Activity in the Details View of the Load Test Analyzer.

The amount of space that is required in the load test results repository to store the timing details data can become very large, especially for longer running load tests. Also, the time to store this data in the load test results repository at the end of the load test is longer because this data is stored on the load test agents until the load test has finished executing. If this is an issue for your testing environment, you might want to set the Timing Details Storage to None.

For more information, see How to: Specify the Timing Details Storage Property for a Load Test Run Setting.

A: Yes, when Visual Studio Ultimate is installed, the load test results store is set up to use an instance of SQL Express that is installed on the computer. SQL Express is limited to using a maximum of 4 GB of disk space. If you are going to run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product. For more information, see Managing Load Test Results in the Load Test Results Repository.

A: Yes, You create scenarios as you step through the New Load Test Wizard. However, any of the settings that you enter in wizard can be modified in the Load Test Editor after the wizard is finished. The Load Test Editor lets you modify an existing scenario, or to add additional scenarios to the load test.

A: Yes.

  1. Open a load test.

  2. In the Load Test Editor, right-click the load test and then choose Add Scenario.

    The Add Scenario Wizard appears. The Add Scenario Wizard is a subset of the New Load Test Wizard you used to create your load test. It contains default values for many of the scenario’s settings. You can keep these default values or change them to values more appropriate for the purposes of your load test.

  3. In each page of this wizard, configure the settings for how you want the scenario to run. For example, specify the load pattern, test mix, browser mix, and network mix. See Create and run a load test.

  4. Choose Finish when you are finished making changes.

    The scenario’s settings are saved. You can view individual settings of existing scenarios in the Load Test Editor, where you can also make changes without using the Add Scenario Wizard.

A: In the full version of Visual Studio Ultimate, the number of virtual users is unlimited. However, if you need to emulate thousands of users, use test controller and test agent machines.

In Visual Studio Ultimate Trial version, the virtual user count is limited to 250.

A: Yes, the open and manage results button Manage results toolbar button in the load test editor. You can have multiple tests open at the same time to compare runs, and create trend analysis reports comparing them. .

A: Yes, these are the differences:

  • Performance counters   A smaller subset of the performance counter data is available while a test is running.

  • Views   When the load test run has completed, the Summary View and Details View are available.

A: Yes, you can specify think times to simulate the time spent by a user on a web page.

A: If you don’t want to set up machines for load testing, or you don’t have available resources, you can use the cloud-based load testing service. It sets up virtual machines in the cloud that will run your load test. Note that your web site must be publicly available on the internet for load testing using Visual Studio Online to access it.

Show:
© 2014 Microsoft