Export (0) Print
Expand All

Stress and Scale Testing

Stress testing is designed to put a system under an unrealistic load. This load level exposes software defects that are only seen under extreme conditions. Examples of these defects are memory leaks and thread contention. Scale testing determines the system configuration that is required to support the anticipated load. It also verifies that the system performs correctly for a large number of users and transactions.

The typical steps for stress and scale testing are the following:

  1. Define the scenarios to test.
  2. Define the counters.
  3. Create the load data and test cases.

Defining the Scenarios to Test

The first step is to identify the scenarios that exercise most of the functionality of the reference implementation. For example, tests for the Partner Portal application include the following scenarios:

  • Browsing the product catalog
  • Browsing promotion pages
  • Read/write requests for incident handling
  • Read/write requests for order exceptions
  • Handling exceptions
  • Typical SharePoint usage

Functional scenarios can be individually executed in isolation or they can be mixed during stress and scale testing. When you mix scenarios, each scenario's frequency of execution should be weighted to reflect real-world load patterns. For example, the catalog browsing scenario should occur more frequently than exception handling.

To accurately define a scenario, you need to analyze the dataset and conditions required to create the load on the application. For example, in the Product Catalog scenario, the test team created a dataset of 100,000 products.

A potential performance concern in the design of the Partner Portal application was the use of item-level permissions in large lists. Therefore, a test was created that included 100 partners with 10 promotion pages each. The test showed that the use of item-level permissions did not degrade performance significantly.

Defining the Counters

Counters measure system parameters while the tests execute. You must identify the counters that will monitor the servers during the stress and scale tests. The following table contains a subset of the counters that were used to analyze the performance of the Partner Portal scenario.

Partner Portal Performance Counters

Test result area

Category

Counters

Computer

Comment

Overall

Test

Passed tests

Failed tests

Tests/second

Application machine(s) and Web front-ends

This category contains counters for test results and test rates. These are the most important counters.

Computers

.NET CLR exceptions

Contentions/second

Application machine(s) and Web front-ends

.NET exceptions thrown per second.

Computers

.NET CLR

LocksAndThreads

Contentions/second

Application machine(s) and Web front-ends

.NET thread contentions thrown per second.

Computers

Process

W3WP

Application machine(s) and Web front-ends

Process associated with the Application Pool. Typically, it allocates the amount of a resource. This helps your computer to be more stable and secure.

Computers

Processor

Percentage of processor time

Application machine(s) and Web front-ends

Total process cycles being used on the machine.

Computers

Memory

Available MBs

Application machine(s) and Web front-ends

Total free memory available; make sure that it is stable during the test.

Computers

.NET CLR memory

Gen 0 collections

Gen 1 collections

Gen 2 collections

Application machine(s) and Web front-ends

Examine the Gen number to make sure that the relationship between Gen 0, Gen 1, and Gen 2 are balanced.

Computers

ASP.NET applications

Requests/second

Application machine(s) and Web front-ends

Measures the requests/second to determine the throughput of the application.

Creating the Load Data and Test Cases

Creating the load data and the test cases is a significant portion of the testing effort. You must complete the following tasks during this step:

  1. Develop utilities to create the load data set.
  2. Create test cases that represent the scenarios that you previously identified.
  3. Create the load tests, which bind the test cases to the load data and then execute the tests.

Typically, you have to automate the creation of load data because of the amount of data required. For example, the developers of the Partner Portal application created the following load data for stress and load testing:

  • A set of 100,000 products in the catalog accessed by the Partner Portal.
  • 100 partner sites, each with 10 users for each partner.
  • 1000 randomly generated promotions for products, with permissions for each promotion assigned to a specific partner.
  • Comma-separated value (CSV) data files that provide the test cases with data to bind to for each submission.

Between runs testers reset the agents and any initial conditions that the tests required.

Creating Web Tests

Web tests are tests that simulate user-browser interactions with SharePoint. For the Partner Portal example, the Visual Studio record and playback engine was used to create the scale tests. The recorded scripts were modified to make the tests more flexible and resilient; for example, they use dynamic variables. The modifications addressed issues such as the use of GUIDs in URLs and pages that change for each site instance created by SharePoint. To gain a deeper understanding of the approach, see How To: Create Automated UI Test Methods for SharePoint. For more information about the record and playback engine in Visual Studio, see Introduction to Record and Playback Engine on MSDN.

After the Web test is parameterized, it is bound to a dataset or tables that are used to drive the test. For example, for the browsing the product catalog scenario, the tests include a CSV file that contains 100,000 rows of product information that test case required so that it could execute. The test case can bind to the CSV to sequentially or randomly access this dataset, depending on how the test is configured.

The next topics explain the following:

  • Stress Testing. The goal of stress testing is to run a system at abnormally high load levels to identify issues such as memory leaks, thread contention, failure modes, and bottlenecks.
  • Scale Testing. Scale testing determines the capacity of a target environment that is running an application so that you can adjust the system for the expected throughput.

Home page on MSDN | Community site

Show:
© 2015 Microsoft