Integration Testing for Application Blocks

 

Microsoft Corporation

January 2005

Summary: An in-depth look at how to test application blocks that have been customized or integrated with other applications.

Contents

Objectives

Overview

Unit Testing a Customized Application Block

Integration Testing an Application Block

Summary

Appendix

Objectives

Unit test a customized application block.

Integration test a customized application block.

Overview

Microsoft application blocks are designed, developed, and tested extensively so that they can be integrated with your applications in an off-the-shelf manner, without any further modification or testing. If you need to modify or extend the existing functionality to suit your application requirements, you may need to unit test the application block before integrating it with your application. There are several benefits to customizing an application block, including the following:

• You can add functionality to the application block so that it is able to fulfill application-specific needs. For example, if the application block supports Microsoft® SQL Server™ 2000, you may need to write custom code to support Oracle databases.
• You can modify the design or code to suit the deployment-specific requirements. For example, you could change the permission and link demand settings in the application block code to suit the application requirements.
• You can extend the interfaces exposed by the application block to write custom implementations, rather than using the implementation provided by application block.

After the application block is customized, you need to unit test the changes to make sure that the customizations do not break existing functionality. You also need to make sure that the changes adhere to the requirements of the application that the application block is to be integrated with.

Whether the application block is customized or not, the integration of the application block with the application needs to be tested to ensure that the application block handles all input from the application and meets all performance and security objectives.

For the purpose of illustration, this chapter refers to a fictitious application and an existing application block. The application requires customization of the application block before integration.

Sample Application Block

The application block referred to throughout this chapter is the Configuration Management Application Block (CMAB). This chapter assumes that the CMAB has already been through test iterations and is ready to be used without modification. The CMAB has the following features:

• It provides the functionality to read and store configuration information transparently in persistent storage media. The storage media are SQL Server, the registry, and an XML file.
• It provides a configurable option to store the information in encrypted form and plain text using XML notation.
• It can be used with desktop applications and Web applications that are deployed in a Web farm.
• It caches configuration information in memory to reduce cross-process communication such as reading from any persistent medium. This caching reduces the response time of the request for any configuration information. The expiration and scavenging mechanism for the data that is cached in memory is similar to the CRON algorithm in UNIX.
• It can store and return data from various locales and cultures without any loss of data integrity.

Sample Business Application

For the purpose of illustration, this chapter uses a Web application that needs to use the CMAB to access configuration-related information. The Web application must customize the CMAB to access configuration data from an Oracle database because the CMAB does not support this type of database off-the-shelf. This section describes the details of the Web application and its use of the CMAB. The business-to-customer Web application is an online bookstore that acts as a liaison between the publishers and the customers. The business process involved and the players in the business process are shown in Figure 9.1.

Figure 9.1. Online bookstore sample application

The sample application lists the book titles, which are categorized according to the publisher or the type of book (such as fiction or nonfiction). The user can use either the Browse or the Search feature to search for a particular title. Based on the information the user sees, the user can buy a soft copy (e-book) or a hard copy (print edition) of the book. The user must be a registered user to buy a book. To complete the purchase, the online bookstore contacts the publishers and the banks through their respective Web services. The online bookstore currently targets U.S.-based customers only, but it needs globalization support so that it can extend its target audience in the future.

The use cases for the online bookstore sample application integrated with the CMAB are as follows:

• Registering with the Web site. Users must enter their profile information when registering. The profile information is stored in a database.
• Personalization. Users can personalize the Web page templates and the content to be displayed. Users can personalize a variety of information, as follows:
  • Users can choose menu color, style, and positions.
  • Users can choose the number of titles listed per page.
  • Users can choose information to be listed with each title, such as author, an abstract of the book, and ISBN.
  • Users can choose how books are categorized, such as by publisher or subject.
  • Users can choose to list titles from particular publishers only.
  • Users can choose the method of payment.
• Browse. When a user logs on and browses through the listed titles, the titles and other information are displayed based on the user's personalized information. Advertisements for new titles are displayed based on the historical information for each user.
• Search. The advertisements for new titles are displayed based on the title that the user searches for.
• Order the soft copy. When the user buys a soft copy of a book, after the purchase is made, an event notification is sent to the Web services for the publisher and the bank.
• Order the hard copy. When the user orders a hard copy of the book from the publisher, after making the payment, an event notification is sent to the Web services for the publisher and the bank.

The online bookstore sample application has a considerable amount of information to be configured. This information is managed transparently using the CMAB. The configuration information can be categorized into user-based configuration information and administrator-controlled configuration information.

The user-based configuration information consists of the following:

• User interface personalization information. The user interface personalization information consists of the template for the Web pages and the content to be displayed. This information is stored in the "Personalization" use case and is retrieved in all other use cases through the CMAB.
• Usage pattern-based personalized information. The historical usage pattern information is stored during execution of the "Browse" and "Order..." use cases, and is retrieved during the browsing or searching scenarios through the CMAB. This information is used to display news and advertisements for new titles.

The administrator-controlled configuration information consists of the following:

• URLs of the Web services. The URLs of the Web services for the publisher and the bank are stored in a persistent medium and are retrieved using the CMAB in the use cases described earlier.
• Database connection strings. A database connection string is used to store and retrieve user profile information from the database; therefore, this information is always required. The data is stored in encrypted form.
• Sinks where information, warnings, and exceptions have to be logged. The settings for the event sinks are retrieved using the CMAB. Event sinks are used for logging events of different categories (such as information, warnings, exceptions, and so on).
• Different authorization providers for each bank or payment mode. The details and settings about the authorization providers are retrieved from the configuration data store—using the CMAB—when a payment is made by the user either by providing the bank account details or credit card details in the "Order..." use cases.
• Business event settings that generate notification events. These are the details and the settings of the business events that generate notifications that are sent to the Web services when buying or ordering actions are complete.

The preceding configuration information is managed (stored, retrieved, and cached) using the CMAB. The data store to be used for storing configuration data for the sample application is an Oracle database. Because the CMAB is not packaged with the support for reading and writing data to and from an Oracle data store, you must add this functionality to the CMAB by implementing the exposed interfaces. You can add the functionality for reading and writing data to and from an Oracle data store by adding a customized custom storage provider assembly to the CMAB. You then need to unit test the customized CMAB to ensure that the changes function properly.

Unit Testing a Customized Application Block

Application blocks may need to be customized based on the requirements of the application. After the application block is customized, it must be unit tested to make sure that the customizations do not break the functionality of existing features. Unit testing also ensures that the customizations are in accordance with the requirements of the application that the block is to be integrated with.

For example, in the case of the online bookstore sample application, the CMAB is required to read and write configuration data to and from an Oracle data store. Therefore, the CMAB has to be customized to manage configuration data stored in an Oracle database. The unit testing is based on the process described in Chapter 3, "Testing Process for Application Blocks."

The following are required for planning the unit tests for the customized application block:

• Requirements and functional specifications for the CMAB and the customizations to be made to the CMAB.
• Performance targets for the sample application use cases using the CMAB.
• Deployment architecture for the sample application.

The unit testing process for the CMAB includes the following steps:

1. Create test plans.
2. Review the design.
3. Review the implementation.
4. Perform black box testing.
5. Perform white box testing.
6. Regression test the existing functionality.

The next sections describe each of these steps.

Step 1: Create Test Plans

The test plan and the test cases include all scenarios that involve the custom storage provider for Oracle. The test plan and test cases should include cases related to design review, code review, white box testing, load testing, stress testing, globalization testing, and security testing. Table 9.1 shows a sample test plan, and Table 9.2 shows a sample test case. The steps in Table 9.2 summarize the test cases for various aspects for which test plans need to be created.

Table 9.1: Sample Test Plan from the Detailed Test Plan Document for the CMAB

Table 9.2: Sample Test Case from the Detailed Test Case Document for the CMAB

Step 2: Review the Design

Review the design of the custom storage provider for Oracle to ensure that it addresses all functional requirements. There are various other important review criteria that you should verify during the review. For more information about these criteria, see Chapter 4, "Design Review for Application Blocks." Table 9.3 summarizes the test cases for the design review of the custom storage provider for an Oracle database.

Table 9.3: Sample Design Review–related Test Cases from the Detailed Test Plan Document for the CMAB

Step 3: Review the Implementation

Review the code for the functionality implemented for the Oracle custom storage provider. While reviewing the code, check whether naming standards have been followed, comments are consistent with the other application block modules, loops and conditional statements are optimized for better performance, security requirements are met when connecting to an Oracle database, and exception handling is consistent with the other storage provider modules. The review criteria are based on the processes described in Chapter 5, "Code Review for Application Blocks." Table 9.4 summarizes the test cases for code review of the implementation of the custom storage provider for an Oracle database.

Table 9.4: Sample Code Review–related Test Cases from the Detailed Test Plan Document for the CMAB

Step 4: Perform Black Box Testing

The black box testing ensures that the API exposed by the custom storage provider for an Oracle database consistently provides the expected output for various types of application-specific input. You need to develop a custom test suite, such as the NUnit suite, or dummy applications to test the module for various types of input. The input could include data such as the user's personalization information, the URLs that need to be retrieved for various Web services, or the business event settings.

The custom storage provider should also be load tested to verify that it is able to handle normal and peak loads as defined in the performance objectives of the sample application. You need to conduct a series of performance tests to ensure that there are no memory leaks and no contentions when the application block is load tested for long durations.

There are other steps that need to be executed during the black box testing phase of the application block. For more information about the additional steps, see Chapter 6, "Black Box and White Box Testing for Application Blocks." Table 9.5 summarizes the test cases for black box testing of the custom storage provider for an Oracle database.

Table 9.5: Sample Black Box Test–related Test Cases from the Detailed Test Plan Document for the CMAB

Step 5: Perform White Box Testing

White box testing involves analyzing the code and identifying the internal functions, such as a particular code path that needs to be tested for various inputs. White box testing also involves profiling the code from various perspectives. Table 9.6 summarizes the test cases for white box testing of the first iteration of a custom storage provider for an Oracle database. The table may be updated based on the inputs from code review and black box testing phases, during which there may be evidence of code that needs to be further analyzed and tested. For detailed information about white box testing, see Chapter 6, "Black Box and White Box Testing for Application Blocks."

Table 9.6: Sample White Box Test–related Test Cases from the Detailed Test Plan Document for the CMAB

Step 6: Regression Test the Existing Functionality

After you ensure that the newly added functionality is working as expected and has passed the test criteria, you should regress the existing functionality to ensure that the modifications have not broken the existing code. You can do this by running the test suites created for the application block. These NUnit tests were also used during the development of the application block.

The test case from the detailed test plan document that corresponds to regression testing is shown in Table 9.7.

Table 9.7: Sample Regression Test–related Test Cases from the Detailed Test Plan Document for the CMAB

After the custom storage provider for Oracle passes the unit testing phase, it is integrated with the sample application. At that point, integration testing is done to ensure that the CMAB addresses all of the requirements of the application.

More Information

For more information about unit testing a customized application block, see the following resources:

• For more information about how to conduct design review for the application block, see Chapter 4, "Design Review for Application Blocks."
• For more information about how to conduct code review for the application block, see Chapter 5, "Code Review for Application Blocks."
• For more information about how to perform functionality testing for the application block, see Chapter 6, "Black Box and White Box Testing for Application Blocks."
• For more information about how to perform performance testing for the application block, see Chapter 8, "Performance Testing for Application Blocks."
• For more information about how to perform globalization testing for the application block, see Chapter 7, "Globalization Testing for Application Blocks."

Integration Testing an Application Block

Integration testing is the logical extension of unit testing. You may be using the application block as packaged, or you may have customized it before integrating it in your application. In either case, you need to conduct integration testing to ensure that the application block can meet the application-specific requirements. Integration testing ensures the following:

• The interfaces between the integrated units of the application and the application block are able to interact in accordance with the specifications.
• The modules of the application with which the application block has been integrated meet all of the functional requirements, performance objectives, globalization objectives, security objectives, and so on.

Integration testing is important because a piece of code that functions correctly when it is tested as a separate unit can demonstrate problems when it is integrated into the actual application. For example, an incompatible data type could be provided by the application to the application block, or data storage could be locked by some part of the application. In the worst scenario, you may discover issues such as the application block being unable to meet the performance objectives of the application or a loss of data at particular times, such as when the block passes decimals with accuracy to 10 digits of the decimal.

However, in most situations you will discover issues such as improper invocation of the interfaces exposed by the application block, missing error handling, failure to trap all of the exceptions thrown by the application block, incorrect data in configuration files, or mismatch of data types leading to loss of data. The majority of the errors that are discovered during the integration phase of the application block occur at the interface between the application block and the application.

Note that the integration testing of the application block is part of the overall testing process for the application itself. The application may also be integrating the other modules, and the following process is not a substitute for any other kind of testing that needs to be done for the application. The following process focuses on testing the integration of the application block with the application.

For the purposes of illustration, the CMAB is assumed to be integrated with the sample application, which requires configuration data to be stored in an Oracle database. Therefore, a custom storage provider has to be added to the CMAB for reading and writing configuration data from and to an Oracle database. The sample application allows users to personalize the Web site and order a book online.

Input for Integration Testing

The following are required for integration testing of an application block:

• Functional specifications of the online bookstore sample application and application block
• Requirements for the sample application and application block
• Performance objectives for the sample application
• Deployment scenarios for the sample application

Steps for Integration Testing

The integration testing process for an application block is shown in Figure 9.2.

Figure 9.2. The integration testing process

The integration testing process includes the following steps:

1. Create test plans. Create the test plans for integration testing of the application block with the application.
2. Perform a code review of the application modules that integrate the application block. The code review helps to ensure that the application code correctly invokes the application block's API.
3. Execute the use cases of the application. These are the use cases that consume the application block during their execution.
4. Perform load testing. Execute load testing–related test cases for the use cases that integrate the application block.
5. Perform stress testing. Execute stress testing–related test cases for the use cases that integrate the application block.
6. Perform globalization testing. Execute globalization testing–related test cases for the use cases that integrate the application block.
7. Perform security testing. Execute security testing–related test cases for the use cases that integrate the application block.

Step 1: Create Test Plans

You can create the detailed test plan and the detailed test case documents for testing the integration of the application block with the application even before the actual integration of the application block is completed. These documents are based on the functional specifications of the application modules that will use the application block. The detailed test plan needs to consider all of the possible use cases, the exceptional flows of the use cases, and testing of use cases with various input types (both valid and invalid input). The detailed test plan also includes test cases for ensuring that the integrated modules satisfy the requirements related to performance, security, globalization, and so on.

For the online bookstore sample application, the detailed test plan and detailed test case should consider all scenarios that involve the CMAB; modules that the sample application uses to store and retrieve the configuration information, such as the personalized user information; possible input to the CMAB that includes input from different locales; and possible return values from the CMAB to the sample application, such as the menu information or URLs of the Web services.

Step 2: Perform Code Review of the Application Modules That Integrate the Application Block

You should perform the code review of the modules that integrate the application block to ensure the following:

• The configuration settings in the configuration files, if any, are correct and in accordance with the requirements.
• The interfaces of the application block are invoked as intended with the expected data types.
• All of the specific exceptions that are thrown by the application block are properly captured and acted upon in the application. This may include logging them to specific sinks or showing appropriately formatted messages to the users. The application block may use the inner exception object for additional information about the exception; the code review should ensure that, where applicable, the information from the inner exception object is also captured.

Note that the scope of the code review is limited to usage of the application block. A thorough code review of the application modules is part of the testing process for the application itself.

Table 9.8 summarizes the test cases for the code review of the online bookstore sample application modules that use the CMAB.

Table 9.8: Sample Integration Test-related Test Cases from the Detailed Test Plan Document for the Online Bookstore Sample Application

Step 3: Execute the Use Cases of the Application

After implementing the code required for application block integration, you need to test the functionality of the use cases related to the application block. The functional testing involves the following:

1. Test the normal flow of the use cases, which includes all possible combinations that each use case can be executed in. This involves performing various user actions by providing the expected input within the specified range and may include the boundary input.
2. Test the exception flow by providing invalid input or conditions that generate exceptions, such as shutting down the Oracle database server during the use case execution or providing incorrect configuration details in the configuration file.

These are the essential test cases related to the testing of application-specific functionality. You will always have these test cases, irrespective of whether or not you are integrating a module with an application. These test cases ensure that the application is functioning as expected.

For the sample application, you must test the use cases related to the CMAB. The test cases are summarized in Table 9.9. The detailed test cases for each test case from Table 9.9 are summarized in Table 9.19 in the "Appendix" section of this chapter.

You may need to make temporary changes to the application module code to simulate invalid input or exceptional conditions. In most scenarios, you will not need to develop test suites because all input will be provided through the application by user actions or background processes.

Table 9.9: Sample Functional Test–related Test Cases from the Detailed Test Plan Document for the Online Bookstore Sample Application

Table 9.10 summarizes the expected results for execution of each test case from Table 9.19 (listed in the "Appendix" section later in this chapter). You should note that Table 9.19 lists only one of the possible flow paths for the use case. If the use case involves any alternate flow paths, you should test all of the flow paths with different combinations of input data.

Table 9.10: Sample Expected Results for the Online Bookstore Sample Application

Step 3: Perform Load Testing

Load testing of the integrated application is required to analyze its behavior at various load levels. Load testing helps to verify that the application meets the performance objectives required for the application after integrating the application block. Because of the integration, there may be bottlenecks in the application—such as resource contention with other modules—that could reduce application performance. Load testing helps to identify such bottlenecks that usually surface only when the application is subjected to a load of concurrent users.

The test cases are the same test cases that you would execute for the application, whether or not you are integrating an application block.

The steps to perform load testing follow the load testing process described in Chapter 16, "Testing .NET Application Performance" of Improving .NET Application Performance and Scalability on MSDN at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetchapt16.asp.

The steps to perform load testing are the following:

1. Identify the test scenarios.
2. Identify the workload.
3. Identify the metrics.
4. Create the test cases.
5. Simulate the load.
6. Analyze the results.

The next sections describe each of these steps as they apply to load testing.

Identify the Test Scenarios

The inputs for this step can be obtained from the following sources:

• Functional specifications or usage scenarios of the online bookstore sample application and application block
• Requirements for the sample application and application block
• Performance objectives for the sample application
• Performance model

The selected usage scenarios of the application are ones that integrate and use application block.

Consider the online bookstore sample application. The CMAB has been integrated to manage the configuration information for this application. The sample usage scenarios are listed in Table 9.19 in the "Appendix" section later in this chapter.

Identify the Workload

The second step in the process of load testing is the identification of workload. The input for the workload identification is taken from the performance model, marketing data, requirements, and the test scenarios identified in the preceding step. The key components of a workload model for the application are as follows:

• The maximum expected number of simultaneous users logged on to the application. These users have active connections to the Web site; however, all of them may not be sending requests at any given time. The excepted load for the online bookstore sample application is 1000 simultaneous users.
• Various user profiles. In a real-world environment, a user might log on, personalize a Web page, search or browse for items, and then purchase the books. The important point is that each user does multiple operations. Therefore, it is important to have various user profiles and to find out the appropriate distribution of the maximum load among these user profiles. Each user profile may have one or more dominant use cases that the user of that profile most frequently requests, as compared to others.

  In the case of the online bookstore sample application, the typical operations are as follows:

  • Log on
  • Register
  • Personalize
  • Browse
  • Search
  • Buy soft copy
  • Buy hard copy

  Table 9.11 shows the operations that constitute a Browse user profile. Ideally you would study the existing profiles of similar existing applications or conduct market research to develop suitable user profiles for your application.

  Table 9.11: Sample Browse User Profile for Online Bookstore Sample Application

Profiles for the sample application are the following:

• Register profile
• Personalize profile
• Browse profile
• Search profile
• Buy Soft Copy profile
• Buy Hard Copy profile

  At any given time, a Web application has users from various profiles logging on. Therefore, the workload for an application consists of a user profile mix, with each profile constituting a percentage of the total users logged on. Table 9.12 shows the user profile mix for the sample application.

  Table 9.12: Sample User Profile Mix for Online Bookstore Sample Application

• Think time between requests. Think time is the time that a user spends between two consecutive requests. You should estimate the think time based on the complexity of the Web page. The think time needs to be higher if the user needs to fill in personal details, as is the case with "Registering with the Web site" use case. Based on the complexity, calculate the average think time for each of the requested pages. The think time can be anywhere from 5 seconds to more than a minute.

Identify the Metrics

The third step in the process of load testing is the identification of metrics. Metrics are used to determine quantitatively whether the performance goals specified in the requirements are being met and are also used to identify any bottlenecks affecting the performance and scalability of the application. Metrics related to performance goals may be available from System Monitor, or you may need to instrument the code to capture custom metrics. For example, the online bookstore sample application has performance goals for the following, which require custom instrumentation of the code:

• Average number of read operations from the Oracle data store per second
• Average number of write operations to the Oracle data store per second
• Average time for a title search to return results
• Average time for a user to log on
• Average time to submit the personalized information

Metrics in the initial iterations generally give a coarse-grained picture of the overall behavior of application. During subsequent iterations, you can add more metrics to focus on a specific area for a fine-grained picture. For example, if you see an increase in private bytes during testing, you can add counters related to various managed heaps.

In the case of the online bookstore sample application, where hundreds of concurrent users store and read personalized information, browse and search product catalogs, and so on, you need to monitor the disk-related counters because there is a possibility of the disk becoming a bottleneck on the server hosting the Oracle data store.

You should monitor a set of counters (shown in Table 9.13) related to resource utilization on all of the servers. These counters help you to analyze the overall resource utilization of the servers and to determine whether any of the resources—such as processor, memory, disk, or network I/O—are a bottleneck.

Table 9.13: Base Set of Metrics for All Servers in the Deployment

Table 9.14 lists the counters that need to be captured during the first iteration of the load testing. You can add more counters based on the analysis in subsequent iterations.

Table 9.14: Metrics to Be Monitored on the Web Tier

Create the Test Cases

The fourth step in the process of load testing the integrated application is to create the test cases. The test cases are created based on the scenarios identified in Step 1 and workload profiles identified in Step 2. You can create test cases for incremental load testing by increasing the load for a given workload profile until the metrics hit their performance threshold, or you can create a test case for analyzing performance for a specific load of users. Table 9.15 shows a test case with a workload profile and a load of 1,000 simultaneous users.

Table 9.15: Sample Test Case for the Online Bookstore Sample Application

Simulate the Load

You can simulate the load by using load simulation tools such as Microsoft Application Center Test (ACT). You should make sure the load generator does not stress the resources (such as processor or memory) on the client computer.

Analyze the Results

The final step in load testing the application is analyzing results. The metrics are analyzed for any bottlenecks that are hurting application performance and to determine whether the application performance is moving toward or away from the performance objectives.

If you encounter any bottlenecks, you need to identify the cause by either conducting a code review of the module or profiling the code using tools from the debugging toolkit for the Microsoft® Windows® operating system.

If the application block is the cause of the bottleneck, you need to unit test and fix the issues in the application block, and then continue with integration testing.

Step 4: Perform Stress Testing

The integrated application needs to be stress tested to evaluate application behavior at loads greater than the peak operating capacity while the resources to process the load are being denied. This type of testing reveals issues that arise only under abnormal loads, such as synchronization issues, loss of data during network congestion, memory leaks, and so on.

To stress test the integrated application block, you need to identify scenarios, workload profiles, metrics, and test cases. You should use input from load testing, such as the test scenarios, peak operating capacities, workload allocations, and so on, as well as code review results to formulate the test cases for stress testing.

The steps for stress testing are as follows:

1. Identify the key scenarios.
2. Identify the workloads.
3. Identify the metrics.
4. Create the test cases.
5. Simulate the load.
6. Analyze the results.

The next sections describe each of these steps in more detail.

Identify the Key Scenarios

The first step in stress testing is to identify the scenarios that are critical to performance of the overall application. This includes operations such as locking, synchronized access to a resource, or extensive disk or network I/O. You can identify these scenarios based on input from load testing. Usually, such scenarios are resource intensive or occur frequently and affect overall performance directly or indirectly.

For example, in the online bookstore sample application, each usage scenario requires different configuration information. If caching is not enabled for the CMAB, the CMAB must repeatedly access the Oracle data store to read and write configuration information. When the number of users increases to very high loads, the number of CMAB requests to the Oracle database also increases significantly. This increase can lead to excessive disk I/O at the database server, which in turn leads to queue buildup at the database server and an increase in response time. The requests on the Web server continue to wait for a response from the database server, and hold on to the threads from the thread pool. This can cause the requests to time out, and the user may receive "time out" or "server too busy" errors.

Another possible scenario deals with enabling caching for the CMAB, and then caching a large amount of data on the Web server. This significantly increases memory utilization on the Web server and, in worst case, may cause the worker process to recycle.

Table 9.16 lists other stress testing scenarios from the sample application.

Table 9.16: Sample Stress Testing Scenarios for the Online Bookstore Sample Application

Identify the Workloads

The second step in stress testing is to identify workload profiles based on the scenarios identified in the preceding step. You can use the same workload profiles from load testing but increase the load for the dominant use case significantly or stress test the use case in isolation. The input for the workload identification includes the peak operating capacities taken from load testing for a given workload profile.

The workload is usually increased from a value beyond the maximum operating capacity. For example, in the case of the test scenario, if the maximum operating capacity is 1,000 users, then the workload can be varied between 1,100 to 2,000 or more users.

Identify the Metrics

Metrics are specific to the test scenarios, and you should select them based on the seriousness and likelihood of potential problems.

For example, in the case of stress testing with a workload and storing a large amount of information in the application block cache, you should monitor the memory-related counters for the system, the ASP.NET worker process, and the base set of counters listed in Table 9.13. Table 9.17 lists the performance counters you would most likely want to monitor for this scenario.

Table 9.17: Metrics to Be Captured for Stress Testing the Online Bookstore Sample Application

Create the Test Cases

Document the test cases using the workload patterns and scenarios that you identified based on the results of the previous steps.

Table 9.18 shows the results for the sample application.

Table 9.18: Test Case Results for the Online Bookstore Sample Application

Simulate the Load

Simulate the load as specified in the test cases by using load simulation tools such as ACT.

Analyze the Results

The final step in stress testing the application is to analyze the results. If you identify any potential bottlenecks caused by the application block rather than the application itself, you need to conduct white box testing of the application block by profiling or debugging the code using various tools, such as debugging tool kit.

More Information

For more information about analyzing the results of stress testing, see the following resources:

• For more information about tools and techniques for debugging, see "Production Debugging for .NET Framework Applications" at:
  http://msdn.microsoft.com/library/en-us/dnbda/html/DBGrm.asp.
• For more information about debugging deadlocks, see "Scenario: Contention or Deadlock Symptoms" in "Production Debugging for .NET Framework Applications" at:
  http://msdn.microsoft.com/library/en-us/dnbda/html/DBGch03.asp.
• For more information about deadlocks, see Microsoft Knowledge Base article 317723, "INFO: What Are Race Conditions and Deadlocks?" at:
  http://support.microsoft.com/default.aspx?scid=kb;en-us;317723.

Step 5: Perform Globalization Testing

One requirement of the online bookstore sample application is that the application should be easily localizable for Japanese and German locales. This means that the application should be able to accept, process, store, and display Japanese and German characters. The resulting localized application is expected to meet the requirements of the functional specifications and the performance objectives of the application.

The test cases and test scenarios executed as part of functional testing also apply for the functionality testing in globalization. The test cases must be tested for both German and Japanese locales and cultures in the case of the sample application.

You will need to set up the test environment, which may include one or both of the following:

• You may need a target version of the operating system with the appropriate language group installed. The targeted language groups in the case of the sample application are Japanese and German.
• You may need a localized build target of the operating system, for example, the Windows operating system with Japanese as the UI language.

After the environment is set up, you need to execute the application use cases to ensure that the localized application functions as expected. Even though the application block has been tested for globalization in the unit testing phase, it is important to verify that the use cases that integrate the application block work correctly.

More Information

For more information about globalization testing, see the following resources:

• For more information about step-by-step globalization testing for integrated components, see "Globalization Step-by-Step" on the Microsoft Global Development and Computing Portal at:
  http://www.microsoft.com/globaldev/getwr/steps/wrguide.mspx.
• For more information about best practices for globalization, see "Best Practices for Developing World-Ready Applications" in the .NET Framework Developer's Guide on MSDN at:
  http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconbestpracticesforglobalapplicationdesign.asp.

Step 6: Perform Security Testing

You should test the security of the sample application after it is integrated with the application block. This testing should focus on identifying any new vulnerabilities that were introduced as a result of the integration. This testing can be done at two levels:

• Black box integration testing. The focus of this testing should be on the application block integration. You should test for unauthorized access to ensure that the application is secure after integration. This type of testing is done by simulating the production environment, and it attempts to find issues that could arise at run time. The test cases that were created during the application block black box testing phase can be used here by modifying them to use the changed interface. You can also use any test cases developed earlier for the application to ensure that the application is secure.
• White box integration testing. The application threat model should be updated to include application block integration. (It is assumed that the sample application already has a threat model in place.) The new threat model should do the following:
  • Update the application architecture diagram with the application block architecture.
  • Identify any new assets, privilege requirements, and resources accessed by the application block.
  • Based on additional information collected earlier, identify any new threats and update the threat model.
  • Prioritize the threats and suggest countermeasures.

Any threats you identify during security testing can be a result of one or both of the following:

• The application block design and/or code did not follow security best practices, such as not supporting encryption if the application block is used for handling sensitive data or not using the best encryption algorithm available to encrypt sensitive data.
• The application block failed to consider specific requirements of the application, such as the application block depending on a service that makes the application vulnerable.

You should also review the integration-related code to ensure that it follows security-related best practices.

More Information

For a detailed description of threats and countermeasures and the process of threat modeling, see Chapter 3, "Threat Modeling" in Improving Web Application Security: Threats and Countermeasures on MSDN at:
http://msdn.microsoft.com/library/en-us/dnnetsec/html/THCMCh03.asp.

Summary

This chapter presented the processes for unit testing the customizations made to the application block and how to test the application block after you integrate it with your application. The process of integration testing assumes that the application block has been rigorously unit tested and meets all requirements of the application, and that the developer has done a careful analysis before deciding to use the application block.

Appendix

Table 9.19: Execution Steps for the Online Bookstore Sample Application Uses Cases

Start | Previous