Performance Testing Guidance for Web Applications

J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea
Microsoft Corporation

September 2007

Objectives

• Understand the key concepts of load testing.
• Learn how to load-test a Web application.

Overview

This chapter explains how to load-test a Web application. Load testing helps to identify the maximum operating capacity of an application as well as any bottlenecks that might interfere with its operating at capacity. The basic approach to performing load testing on a Web application is:

1. Identify the performance-critical scenarios.
2. Identify the workload profile for distributing the entire load among the key scenarios.
3. Identify the metrics that you want to collect in order to verify them against your performance objectives.
4. Design tests to simulate the load.
5. Use tools to implement the load according to the designed tests, and capture the metrics.
6. Analyze the metric data captured during the tests.

By using an iterative testing process, these steps should help you achieve your performance objectives.

There are many reasons for load-testing a Web application. The most basic type of load testing is used to determine the Web application’s behavior under both normal and anticipated peak load conditions. As you begin load testing, it is recommended that you start with a small number of virtual users and then incrementally increase the load from normal to peak. You can then observe how your application performs during this gradually increasing load condition. Eventually, you will cross a threshold limit for your performance objectives. For example, you might continue to increase the load until the server processor utilization reaches 75 percent, or when end-user response times exceed 8 seconds.

How to Use This Chapter

Use this chapter to understand the key concepts of load testing and the steps involved in load-testing Web applications. To get the most from this chapter:

• Use the “Input” and “Output” sections to understand the key inputs for load testing a Web application and the key outcomes of doing so.
• Use the “Approach for Load Testing” section to get an overview of the approach for load testing a Web application, and as quick reference guide for you and your team.
• Use the various steps sections to understand the details of each step involved in load-testing a Web application.

Input

The following are useful inputs for load-testing a Web application:

• Performance-critical usage scenarios
• Workload models
• Performance acceptance criteria
• Performance metrics associated with the acceptance criteria
• Interview feedback from the designer or developer of the Web application
• Interview feedback from end users of the application
• Interview feedback from the operations personnel who will maintain and manage the application

Output

The main outcomes that load testing helps you to accomplish are:

• Updated test plans and test designs for load and performance testing
• Various performance measures such as throughput, response time, and resource utilization
• Potential bottlenecks that need to be analyzed in the white-box testing phase
• The behavior of the application at various load levels

Approach for Load Testing

The following steps are involved in load-testing a Web application:

1. Step 1 - Identify performance acceptance criteria
2. Step 2 - Identify key scenarios
3. Step 3 - Create a workload model
4. Step 4 - Identify the target load levels
5. Step 5 - Identify metrics
6. Step 6 - Design specific tests
7. Step 7 - Run tests
8. Step 8 - Analyze the results

These steps are graphically represented below. The sections that follow discuss each step in detail.

Figure 17.1 Load Testing Steps

Step 1 - Identify Performance Acceptance Criteria

Identifying performance acceptance criteria is most valuable when initiated early in the application’s development life cycle. It is frequently valuable to record the acceptance criteria for your application and store them in a place and format that is available to the entire team for review and comment. Criteria are typically determined by balancing your business, industry, technology, competitive, and user requirements.

Test objectives frequently include the following:

• Response time.  For example, the product catalog must be displayed in less than 3 seconds.
• Throughput.  For example, the system must support 100 transactions per second.
• Resource utilization.  A frequently overlooked aspect is the amount of resources your application is consuming, in terms of processor, memory, disk input output (I/O), and network I/O.
• Maximum user load.  This test objective determines how many users can run on a specific hardware configuration.
• Business related metrics.  This objective is mapped to business volume at normal and peak values; for example, the number of orders or Help desk calls handled at a given time.

Step 2 - Identify Key Scenarios

Scenarios are anticipated user paths that generally incorporate multiple application activities. Key scenarios are those for which you have specific performance goals, those considered to be high-risk, those that are most commonly used, or those with a significant performance impact. The basic steps for identifying key scenarios are.

1. Identify all the scenarios for a Web application. For example, even the most basic e-commerce application must support the following user scenarios:

• • Browse catalog
  • Search for a product
  • Place an order

1. Identify the activities involved in each of the scenarios. For example, a “Place an Order” scenario will include the following activities:

• • Log on to the application.
  • Browse the product catalog.
  • Search for a specific product.
  • Add items to the shopping cart.
  • Validate credit card details and place an order.

1. Identify the scenarios that are most commonly executed or most resource-intensive; these will be the key scenarios used for load testing. For example, in an e-commerce application, browsing a catalog may be the most commonly executed scenario, whereas placing an order may be the most resource-intensive scenario because it accesses the database.

• • The most commonly executed scenarios for an existing Web application can be determined by examining the log files.
  • The most commonly executed scenarios for a new Web application can be obtained from market research, historical data, market trends, and so on.
  • Resource-intensive scenarios can be identified by using design documents or the actual code implementation. The primary resources are:
  • • Processor
    • Memory
    • Disk I/O
    • Network I/O

Once they have been identified, you will use these key scenarios to create workload profiles and to design load tests.

Step 3 - Create a Workload Model

When defining workload distribution, consider the following key points for determining the characteristics for user scenarios:

• A user scenario is defined as a navigational path, including intermediate steps or activities, taken by the user to complete a task. This can also be thought of as a user session.
• A user will typically pause between pages during a session. This is known as user delay or think time.
• A session will have an average duration when viewed across multiple users. It is important to account for this when defining the load levels that will translate into concurrent usage, overlapping users, or user sessions per unit of time.
• Not all scenarios can be performed by a new user, a returning user, or either; know who you expect your primary users to be and test accordingly.

Step 4 - Identify Target Load Levels

Identify the load levels to be applied to the workload distribution(s) identified during the previous step. The purpose of identifying target load levels is to ensure that your tests can be used to predict or compare a variety of production load conditions. The following are common inputs used for determining target load levels:

• Business volume (both current and projected) as it relates to your performance objectives
• Key scenarios
• Distribution of work
• Session characteristics (navigational path, duration, percentage of new users)

By combining the items above, you can determine the remaining details necessary to implement the workload model under a particular target load.

Step 5 - Identify Metrics

There is a virtually unlimited number of metrics that can be collected during a performance test execution. However, collecting too many metrics can make analysis unwieldy as well as negatively impact the application’s actual performance. For these reasons, it is important to identify the metrics that are most relevant to your performance objectives and those that you anticipate will help you to identify bottlenecks. Only well-selected metrics that are analyzed correctly and contextually provide information of value.  

The following are a few suggestions for identifying the metrics that will provide the most valuable information to your project:

• Define questions related to your application performance that can be easily tested.  For example, what is the checkout response time when placing an order? How many orders are placed in a minute? These questions have definite answers.
• With the answers to these questions, determine quality goals for comparison against external expectations.  For example, checkout response time should be 30 seconds, and a maximum of 10 orders should be placed in a minute. The answers are based on market research, historical data, market trends, and so on.
• Identify the metrics.  Using your list of performance-related questions and answers, identify the metrics that provide information related to those questions and answers.
• Identify supporting metrics.  Using the same approach, you can identify lower-level metrics that focus on measuring the performance and identifying the bottlenecks in the system. When identifying low-level metrics, most teams find it valuable to determine a baseline for those metrics under single-user and/or normal load conditions. This helps you determine the acceptable load levels for your application. Baseline values help you analyze your application performance at varying load levels and serve as a starting point for trend analysis across builds or releases.
• Reevaluate the metrics to be collected regularly.  Goals, priorities, risks, and current issues are bound to change over the course of a project. With each of these changes, different metrics may provide more value than the ones that have previously been identified.

Additionally, to evaluate the performance of your application in more detail and to identify potential bottlenecks, it is frequently useful to monitor metrics in the following categories:

• Network-specific metrics.  This set of metrics provides information about the overall health and efficiency of your network, including routers, switches, and gateways.
• System-related metrics.  This set of metrics helps you identify the resource utilization on your server. The resources being utilized are processor, memory, disk I/O, and network I/O.
• Platform-specific metrics.  Platform-specific metrics are related to software that is used to host your application, such as the Microsoft .NET Framework common language runtime (CLR) and ASP.NET-related metrics.
• Application-specific metrics.  These include custom performance counters inserted in your application code to monitor application health and identify performance issues. You might use custom counters to determine the number of concurrent threads waiting to acquire a particular lock, or the number of requests queued to make an outbound call to a Web service.
• Service-level metrics.  These metrics can help to measure overall application throughput and latency, or they might be tied to specific business scenarios.
• Business metrics.  These metrics are indicators of business-related information, such as the number of orders placed in a given timeframe.

Step 6 - Design Specific Tests

Using your scenarios, key metrics, and workload analysis, you can now design specific tests to be conducted. Each test will generally have a different purpose, collect different data, include different scenarios, and have different target load levels. The key is to design tests that will help the team collect the information it needs in order to understand, evaluate, or tune the application.

Points to consider when designing tests include:

• Do not change your test design because the design is difficult to implement in your tool.
• If you cannot implement your test as designed, ensure that you record the details pertaining to the test that you do implement.
• Ensure that the model contains all of the supplementary data needed to create the actual test.
• Consider including invalid data in your performance tests. For example, include some users who mistype their password on the first attempt but get it correct on a second try.
• First-time users usually spend significantly more time on each page or activity than experienced users.
• The best possible test data is test data collected from a production database or log file.
• Think about nonhuman system users and batch processes as well as end users. For example, there might be a batch process that runs to update the status of orders while users are performing activities on the site. In this situation, you would need to account for those processes because they might be consuming resources.
• Do not get overly caught up in striving for perfection, and do not fall into the trap of oversimplification. In general, it is a good idea to start executing tests when you have a reasonable test designed and then enhance the design incrementally while collecting results.

Step 7 - Run Tests

Poor load simulations can render all of the work in the previous activities useless. To understand the data collected from a test execution, the load simulation must reflect the test design. When the simulation does not reflect the test design, the results are prone to misinterpretation. Consider the following steps when preparing to simulate load:

1. Configure the test environment in such a way that it mirrors your production environment as closely as possible, noting and accounting for all differences between the two.
2. Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.
3. Use appropriate load-generation tools to create a load with the characteristics specified in your test design.
4. Using the load-generation tool(s), execute tests by first building up to the target load specified in your test design, in order to validate the correctness of the simulation. Some things to consider during test execution include:

• • Begin load testing with a small number of users distributed against your user profile, and then incrementally increase the load. It is important to allow time for the system to stabilize between increases in load while evaluating the correctness of the simulation.
  • Consider continuing to increase the load and record the behavior until you reach the threshold for the resources identified in your performance objectives, even if that load is beyond the target load specified in the test design. Information about when the system crosses identified thresholds is just as important as the value of the metrics at the target load of the test.
  • Similarly, it is frequently valuable to continue to increase the number of users until you run up against the service-level limits beyond which you would be violating your SLAs for throughput, response time, and resource utilization.

Note:  Make sure that the client computers (agents) you use to generate load are not overly stressed. Resource utilization such as processor and memory must remain well below the utilization threshold values to ensure accurate test results.

Step 8 - Analyze the Results

You can analyze the test results to find performance bottlenecks between each test run or after all testing has been completed. Analyzing the results correctly requires training and experience with graphing correlated response time and system data.

The following are the steps for analyzing the data:

1. Analyze the captured data and compare the results against the metric’s accepted level to determine whether the performance of the application being tested shows a trend toward or away from the performance objectives.
2. Analyze the measured metrics to diagnose potential bottlenecks. Based on the analysis, if required, capture additional metrics in subsequent test cycles. For example, suppose that during the first iteration of load tests, the process shows a marked increase in memory consumption, indicating a possible memory leak. In the subsequent iterations, additional memory counters related to generations can be captured to study the memory allocation pattern for the application.

Summary

Load testing helps to identify the maximum operating capacity of the application and any bottlenecks that might be degrading performance.

The basic methodology for performing load testing on a Web application is to identify the performance-critical key scenarios; identify the workload profile for distributing all the load among the key scenarios; identify metrics that you want to collect in order to verify them against your performance objectives; create test cases that will be used to simulate the load test; use tools to simulate the load according to the test cases and capture the metrics; and finally, analyze the metrics data captured during the tests.