Integration Testing for Application Blocks
This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.
Summary: An in-depth look at how to test application blocks that have been customized or integrated with other applications.
Unit test a customized application block.
Integration test a customized application block.
Microsoft application blocks are designed, developed, and tested extensively so that they can be integrated with your applications in an off-the-shelf manner, without any further modification or testing. If you need to modify or extend the existing functionality to suit your application requirements, you may need to unit test the application block before integrating it with your application. There are several benefits to customizing an application block, including the following:
- You can add functionality to the application block so that it is able to fulfill application-specific needs. For example, if the application block supports Microsoft® SQL Server 2000, you may need to write custom code to support Oracle databases.
- You can modify the design or code to suit the deployment-specific requirements. For example, you could change the permission and link demand settings in the application block code to suit the application requirements.
- You can extend the interfaces exposed by the application block to write custom implementations, rather than using the implementation provided by application block.
After the application block is customized, you need to unit test the changes to make sure that the customizations do not break existing functionality. You also need to make sure that the changes adhere to the requirements of the application that the application block is to be integrated with.
Whether the application block is customized or not, the integration of the application block with the application needs to be tested to ensure that the application block handles all input from the application and meets all performance and security objectives.
For the purpose of illustration, this chapter refers to a fictitious application and an existing application block. The application requires customization of the application block before integration.
Sample Application Block
The application block referred to throughout this chapter is the Configuration Management Application Block (CMAB). This chapter assumes that the CMAB has already been through test iterations and is ready to be used without modification. The CMAB has the following features:
- It provides the functionality to read and store configuration information transparently in persistent storage media. The storage media are SQL Server, the registry, and an XML file.
- It provides a configurable option to store the information in encrypted form and plain text using XML notation.
- It can be used with desktop applications and Web applications that are deployed in a Web farm.
- It caches configuration information in memory to reduce cross-process communication such as reading from any persistent medium. This caching reduces the response time of the request for any configuration information. The expiration and scavenging mechanism for the data that is cached in memory is similar to the CRON algorithm in UNIX.
- It can store and return data from various locales and cultures without any loss of data integrity.
Sample Business Application
For the purpose of illustration, this chapter uses a Web application that needs to use the CMAB to access configuration-related information. The Web application must customize the CMAB to access configuration data from an Oracle database because the CMAB does not support this type of database off-the-shelf. This section describes the details of the Web application and its use of the CMAB. The business-to-customer Web application is an online bookstore that acts as a liaison between the publishers and the customers. The business process involved and the players in the business process are shown in Figure 9.1.
Figure 9.1. Online bookstore sample application
The sample application lists the book titles, which are categorized according to the publisher or the type of book (such as fiction or nonfiction). The user can use either the Browse or the Search feature to search for a particular title. Based on the information the user sees, the user can buy a soft copy (e-book) or a hard copy (print edition) of the book. The user must be a registered user to buy a book. To complete the purchase, the online bookstore contacts the publishers and the banks through their respective Web services. The online bookstore currently targets U.S.-based customers only, but it needs globalization support so that it can extend its target audience in the future.
The use cases for the online bookstore sample application integrated with the CMAB are as follows:
- Registering with the Web site. Users must enter their profile information when registering. The profile information is stored in a database.
- Personalization. Users can personalize the Web page templates and the content to be displayed. Users can personalize a variety of information, as follows:
- Users can choose menu color, style, and positions.
- Users can choose the number of titles listed per page.
- Users can choose information to be listed with each title, such as author, an abstract of the book, and ISBN.
- Users can choose how books are categorized, such as by publisher or subject.
- Users can choose to list titles from particular publishers only.
- Users can choose the method of payment.
- Browse. When a user logs on and browses through the listed titles, the titles and other information are displayed based on the user's personalized information. Advertisements for new titles are displayed based on the historical information for each user.
- Search. The advertisements for new titles are displayed based on the title that the user searches for.
- Order the soft copy. When the user buys a soft copy of a book, after the purchase is made, an event notification is sent to the Web services for the publisher and the bank.
- Order the hard copy. When the user orders a hard copy of the book from the publisher, after making the payment, an event notification is sent to the Web services for the publisher and the bank.
The online bookstore sample application has a considerable amount of information to be configured. This information is managed transparently using the CMAB. The configuration information can be categorized into user-based configuration information and administrator-controlled configuration information.
The user-based configuration information consists of the following:
- User interface personalization information. The user interface personalization information consists of the template for the Web pages and the content to be displayed. This information is stored in the "Personalization" use case and is retrieved in all other use cases through the CMAB.
- Usage pattern-based personalized information. The historical usage pattern information is stored during execution of the "Browse" and "Order..." use cases, and is retrieved during the browsing or searching scenarios through the CMAB. This information is used to display news and advertisements for new titles.
The administrator-controlled configuration information consists of the following:
- URLs of the Web services. The URLs of the Web services for the publisher and the bank are stored in a persistent medium and are retrieved using the CMAB in the use cases described earlier.
- Database connection strings. A database connection string is used to store and retrieve user profile information from the database; therefore, this information is always required. The data is stored in encrypted form.
- Sinks where information, warnings, and exceptions have to be logged. The settings for the event sinks are retrieved using the CMAB. Event sinks are used for logging events of different categories (such as information, warnings, exceptions, and so on).
- Different authorization providers for each bank or payment mode. The details and settings about the authorization providers are retrieved from the configuration data storeusing the CMABwhen a payment is made by the user either by providing the bank account details or credit card details in the "Order..." use cases.
- Business event settings that generate notification events. These are the details and the settings of the business events that generate notifications that are sent to the Web services when buying or ordering actions are complete.
The preceding configuration information is managed (stored, retrieved, and cached) using the CMAB. The data store to be used for storing configuration data for the sample application is an Oracle database. Because the CMAB is not packaged with the support for reading and writing data to and from an Oracle data store, you must add this functionality to the CMAB by implementing the exposed interfaces. You can add the functionality for reading and writing data to and from an Oracle data store by adding a customized custom storage provider assembly to the CMAB. You then need to unit test the customized CMAB to ensure that the changes function properly.
Application blocks may need to be customized based on the requirements of the application. After the application block is customized, it must be unit tested to make sure that the customizations do not break the functionality of existing features. Unit testing also ensures that the customizations are in accordance with the requirements of the application that the block is to be integrated with.
For example, in the case of the online bookstore sample application, the CMAB is required to read and write configuration data to and from an Oracle data store. Therefore, the CMAB has to be customized to manage configuration data stored in an Oracle database. The unit testing is based on the process described in Chapter 3, "Testing Process for Application Blocks."
The following are required for planning the unit tests for the customized application block:
- Requirements and functional specifications for the CMAB and the customizations to be made to the CMAB.
- Performance targets for the sample application use cases using the CMAB.
- Deployment architecture for the sample application.
The unit testing process for the CMAB includes the following steps:
- Create test plans.
- Review the design.
- Review the implementation.
- Perform black box testing.
- Perform white box testing.
- Regression test the existing functionality.
The next sections describe each of these steps.
Step 1: Create Test Plans
The test plan and the test cases include all scenarios that involve the custom storage provider for Oracle. The test plan and test cases should include cases related to design review, code review, white box testing, load testing, stress testing, globalization testing, and security testing. Table 9.1 shows a sample test plan, and Table 9.2 shows a sample test case. The steps in Table 9.2 summarize the test cases for various aspects for which test plans need to be created.
Table 9.1: Sample Test Plan from the Detailed Test Plan Document for the CMAB
|Scenario 1||Use the NUnit test suite to test reading and writing data to an Oracle data store through the custom storage provider for the Oracle data store.|
|1.1||High||Use the NUnit test suite to test reading and writing data to an Oracle data store through the custom storage provider for the Oracle data store.|
Table 9.2: Sample Test Case from the Detailed Test Case Document for the CMAB
|Scenario to be tested||Use the NUnit test suite to test reading and writing data to an Oracle database using the custom storage provider.|
|Execution details||1. Create an assembly named ConfigurationManagement.UnitTests. |
2. Create the OraceConfigurationManagement.cs file with the OracleConfigurationManagement class in the assembly that you created in the preceding step. (Note that the static public methods of the ConfigurationManager will be used for the read and write operations.)
3. In the OracleConfigurationManagement class, implement methods that perform the following functions:
Implement a method that reads a configuration section from the Oracle database for all different combinations of settings for encryption and signing.
Implement a method that writes a configuration section to the Oracle database for all the different combinations of settings for encryption and signing.
4. Compile the assembly that you created.
5. Run NUnit on this assembly.
|Data required||Not applicable.|
|Expected results||Not applicable.|
|Actual results||Not applicable.|
|Test passed||Not applicable.|
Step 2: Review the Design
Review the design of the custom storage provider for Oracle to ensure that it addresses all functional requirements. There are various other important review criteria that you should verify during the review. For more information about these criteria, see Chapter 4, "Design Review for Application Blocks." Table 9.3 summarizes the test cases for the design review of the custom storage provider for an Oracle database.
Table 9.3: Sample Design Reviewrelated Test Cases from the Detailed Test Plan Document for the CMAB
|Scenario 2||Conduct the design review of the custom storage provider proposed for the application block.|
|2.1||High||Verify that the design addresses all functional specifications and the requirements for the Oracle custom storage provider.|
|2.2||High||Verify that the design suites the deployment scenario for the online bookstore sample application.|
|2.3||High||Verify that the design makes appropriate tradeoffs based on the best practices for performance and scalability.|
|2.4||High||Conduct threat analysis to verify that the design does not create any security vulnerabilities in the application.|
|2.5||Medium||Verify that the design follows best practices related to ease of maintenance.|
|2.6||Medium||Verify that the design adheres to the globalization-related best practices to ensure smooth localization of the application block.|
|2.7||High||Verify that the design has an exception management strategy that ensures proper propagation of exceptions without repeatedly throwing exceptions.|
Step 3: Review the Implementation
Review the code for the functionality implemented for the Oracle custom storage provider. While reviewing the code, check whether naming standards have been followed, comments are consistent with the other application block modules, loops and conditional statements are optimized for better performance, security requirements are met when connecting to an Oracle database, and exception handling is consistent with the other storage provider modules. The review criteria are based on the processes described in Chapter 5, "Code Review for Application Blocks." Table 9.4 summarizes the test cases for code review of the implementation of the custom storage provider for an Oracle database.
Table 9.4: Sample Code Reviewrelated Test Cases from the Detailed Test Plan Document for the CMAB
|Scenario 3||Conduct a code review of the custom storage provider for an Oracle database.|
|3.1||High||Ensure that the implementation addresses all functional specifications and the requirements for the Oracle custom storage provider.|
|3.2||Medium||Verify that naming standards are followed.|
|3.3||Medium||Verify that commenting guidelines are followed.|
|3.4||High||Verify that the performance and scalability guidelines are followed.|
|3.5||High||Verify that the guidelines for writing secure code are followed.|
|3.6||High||Verify that globalization-related guidelines are followed.|
|3.7||High||Run the FxCop code review tool to ensure that all guidelines supported by the tool are followed in the code.|
|3.8||High||Verify that the extended functionality follows the exception management strategy in the CMAB.|
Step 4: Perform Black Box Testing
The black box testing ensures that the API exposed by the custom storage provider for an Oracle database consistently provides the expected output for various types of application-specific input. You need to develop a custom test suite, such as the NUnit suite, or dummy applications to test the module for various types of input. The input could include data such as the user's personalization information, the URLs that need to be retrieved for various Web services, or the business event settings.
The custom storage provider should also be load tested to verify that it is able to handle normal and peak loads as defined in the performance objectives of the sample application. You need to conduct a series of performance tests to ensure that there are no memory leaks and no contentions when the application block is load tested for long durations.
There are other steps that need to be executed during the black box testing phase of the application block. For more information about the additional steps, see Chapter 6, "Black Box and White Box Testing for Application Blocks." Table 9.5 summarizes the test cases for black box testing of the custom storage provider for an Oracle database.
Table 9.5: Sample Black Box Testrelated Test Cases from the Detailed Test Plan Document for the CMAB
|Scenario 4||Conduct black box testing of the custom storage provider for an Oracle database.|
|4.1||High||Create NUnit tests to test the external interfaces of the custom storage provider for an Oracle database.|
|4.2||High||Test the external interfaces for various types of input that are within the specified range of expected input for an application that integrates the application block.|
|4.3||High||Test the external interfaces for various types of input that are at the boundary of the specified range of expected input for an application that integrates the application block.|
|4.4||High||Test the external interfaces for various types of input that are outside the specified range of expected input for an application that integrates the application block. This input may also be categorized as invalid input.|
|4.5||High||Conduct load tests of the custom storage provider to ensure that it meets the performance targets for the application block.|
|4.6||High||Conduct load tests for an extended duration to ensure that there are no memory leaks.|
|4.7||High||Conduct stress tests by reading and writing large amounts of data on a per-user basis to analyze bandwidth utilization and application block behavior under high loads.|
|4.8||High||Conduct load tests for targeted peak operating capacity to ensure that there is no contention.|
|4.9||High||Conduct stress tests to analyze the behavior of the custom storage provider beyond the targeted peak operating capacity.|
Step 5: Perform White Box Testing
White box testing involves analyzing the code and identifying the internal functions, such as a particular code path that needs to be tested for various inputs. White box testing also involves profiling the code from various perspectives. Table 9.6 summarizes the test cases for white box testing of the first iteration of a custom storage provider for an Oracle database. The table may be updated based on the inputs from code review and black box testing phases, during which there may be evidence of code that needs to be further analyzed and tested. For detailed information about white box testing, see Chapter 6, "Black Box and White Box Testing for Application Blocks."
Table 9.6: Sample White Box Testrelated Test Cases from the Detailed Test Plan Document for the CMAB
|Scenario 5||Conduct white box testing of the custom storage provider for an Oracle database.|
|5.1||High||Run a code coverage tool to ensure that there is no redundant code in the custom storage provider, and that all possible paths in the code can be traversed under various usage scenarios.|
|5.2||High||Analyze the memory allocation pattern of the custom storage provider to ensure that there are no side effect allocations, and that the number of collections for Gen 0, 1, 2 is near the ideal ratio of 1:0.1:0.01.|
|5.3||High||Test internal functions for various types of inputs to ensure that there is no loss of data submitted by the user, and that the results are as expected.|
|5.4||High||Perform security testing by simulating hostile deployment environments after analyzing the code access security.|
Step 6: Regression Test the Existing Functionality
After you ensure that the newly added functionality is working as expected and has passed the test criteria, you should regress the existing functionality to ensure that the modifications have not broken the existing code. You can do this by running the test suites created for the application block. These NUnit tests were also used during the development of the application block.
The test case from the detailed test plan document that corresponds to regression testing is shown in Table 9.7.
Table 9.7: Sample Regression Testrelated Test Cases from the Detailed Test Plan Document for the CMAB
|Scenario 6||Run the existing NUnit tests for the application block to ensure that the existing functionality is working as expected.|
|6.1||Medium||Run the existing NUnit tests for the application block to ensure that the existing functionality is working as expected.|
After the custom storage provider for Oracle passes the unit testing phase, it is integrated with the sample application. At that point, integration testing is done to ensure that the CMAB addresses all of the requirements of the application.
For more information about unit testing a customized application block, see the following resources:
- For more information about how to conduct design review for the application block, see Chapter 4, "Design Review for Application Blocks."
- For more information about how to conduct code review for the application block, see Chapter 5, "Code Review for Application Blocks."
- For more information about how to perform functionality testing for the application block, see Chapter 6, "Black Box and White Box Testing for Application Blocks."
- For more information about how to perform performance testing for the application block, see Chapter 8, "Performance Testing for Application Blocks."
- For more information about how to perform globalization testing for the application block, see Chapter 7, "Globalization Testing for Application Blocks."
Integration testing is the logical extension of unit testing. You may be using the application block as packaged, or you may have customized it before integrating it in your application. In either case, you need to conduct integration testing to ensure that the application block can meet the application-specific requirements. Integration testing ensures the following:
- The interfaces between the integrated units of the application and the application block are able to interact in accordance with the specifications.
- The modules of the application with which the application block has been integrated meet all of the functional requirements, performance objectives, globalization objectives, security objectives, and so on.
Integration testing is important because a piece of code that functions correctly when it is tested as a separate unit can demonstrate problems when it is integrated into the actual application. For example, an incompatible data type could be provided by the application to the application block, or data storage could be locked by some part of the application. In the worst scenario, you may discover issues such as the application block being unable to meet the performance objectives of the application or a loss of data at particular times, such as when the block passes decimals with accuracy to 10 digits of the decimal.
However, in most situations you will discover issues such as improper invocation of the interfaces exposed by the application block, missing error handling, failure to trap all of the exceptions thrown by the application block, incorrect data in configuration files, or mismatch of data types leading to loss of data. The majority of the errors that are discovered during the integration phase of the application block occur at the interface between the application block and the application.
Note that the integration testing of the application block is part of the overall testing process for the application itself. The application may also be integrating the other modules, and the following process is not a substitute for any other kind of testing that needs to be done for the application. The following process focuses on testing the integration of the application block with the application.
For the purposes of illustration, the CMAB is assumed to be integrated with the sample application, which requires configuration data to be stored in an Oracle database. Therefore, a custom storage provider has to be added to the CMAB for reading and writing configuration data from and to an Oracle database. The sample application allows users to personalize the Web site and order a book online.
Input for Integration Testing
The following are required for integration testing of an application block:
- Functional specifications of the online bookstore sample application and application block
- Requirements for the sample application and application block
- Performance objectives for the sample application
- Deployment scenarios for the sample application
Steps for Integration Testing
The integration testing process for an application block is shown in Figure 9.2.
Figure 9.2. The integration testing process
The integration testing process includes the following steps:
- Create test plans. Create the test plans for integration testing of the application block with the application.
- Perform a code review of the application modules that integrate the application block. The code review helps to ensure that the application code correctly invokes the application block's API.
- Execute the use cases of the application. These are the use cases that consume the application block during their execution.
- Perform load testing. Execute load testingrelated test cases for the use cases that integrate the application block.
- Perform stress testing. Execute stress testingrelated test cases for the use cases that integrate the application block.
- Perform globalization testing. Execute globalization testingrelated test cases for the use cases that integrate the application block.
- Perform security testing. Execute security testingrelated test cases for the use cases that integrate the application block.
Step 1: Create Test Plans
You can create the detailed test plan and the detailed test case documents for testing the integration of the application block with the application even before the actual integration of the application block is completed. These documents are based on the functional specifications of the application modules that will use the application block. The detailed test plan needs to consider all of the possible use cases, the exceptional flows of the use cases, and testing of use cases with various input types (both valid and invalid input). The detailed test plan also includes test cases for ensuring that the integrated modules satisfy the requirements related to performance, security, globalization, and so on.
For the online bookstore sample application, the detailed test plan and detailed test case should consider all scenarios that involve the CMAB; modules that the sample application uses to store and retrieve the configuration information, such as the personalized user information; possible input to the CMAB that includes input from different locales; and possible return values from the CMAB to the sample application, such as the menu information or URLs of the Web services.
Step 2: Perform Code Review of the Application Modules That Integrate the Application Block
You should perform the code review of the modules that integrate the application block to ensure the following:
- The configuration settings in the configuration files, if any, are correct and in accordance with the requirements.
- The interfaces of the application block are invoked as intended with the expected data types.
- All of the specific exceptions that are thrown by the application block are properly captured and acted upon in the application. This may include logging them to specific sinks or showing appropriately formatted messages to the users. The application block may use the inner exception object for additional information about the exception; the code review should ensure that, where applicable, the information from the inner exception object is also captured.
Note that the scope of the code review is limited to usage of the application block. A thorough code review of the application modules is part of the testing process for the application itself.
Table 9.8 summarizes the test cases for the code review of the online bookstore sample application modules that use the CMAB.
Table 9.8: Sample Integration Test-related Test Cases from the Detailed Test Plan Document for the Online Bookstore Sample Application
|Scenario 1||Perform a code review of the application modules that integrate the application block. The details of the criteria used for the review will be documented in the detailed test case document.|
|1.1||Medium||Perform code review of the "Registering with the Web Site" use case.|
|1.2||Medium||Perform code review of the "Personalization" use case.|
|1.3||Medium||Perform code review of the "Browse" use case.|
|1.4||Medium||Perform code review of the "Search" use case.|
|1.5||Medium||Perform code review of the "Order the Soft Copy" use case.|
|1.6||Medium||Perform code review of the "Order the Hard Copy" use case.|
Step 3: Execute the Use Cases of the Application
After implementing the code required for application block integration, you need to test the functionality of the use cases related to the application block. The functional testing involves the following:
- Test the normal flow of the use cases, which includes all possible combinations that each use case can be executed in. This involves performing various user actions by providing the expected input within the specified range and may include the boundary input.
- Test the exception flow by providing invalid input or conditions that generate exceptions, such as shutting down the Oracle database server during the use case execution or providing incorrect configuration details in the configuration file.
These are the essential test cases related to the testing of application-specific functionality. You will always have these test cases, irrespective of whether or not you are integrating a module with an application. These test cases ensure that the application is functioning as expected.
For the sample application, you must test the use cases related to the CMAB. The test cases are summarized in Table 9.9. The detailed test cases for each test case from Table 9.9 are summarized in Table 9.19 in the "Appendix" section of this chapter.
You may need to make temporary changes to the application module code to simulate invalid input or exceptional conditions. In most scenarios, you will not need to develop test suites because all input will be provided through the application by user actions or background processes.
Table 9.9: Sample Functional Testrelated Test Cases from the Detailed Test Plan Document for the Online Bookstore Sample Application
|Scenario 2||Test the functionality of the use cases of the online bookstore sample application, which integrates the CMAB.|
|2.1||High||Test the "Registering with the Web Site" use case.|
|2.2||High||Test the "Personalization" use case for personalization of Web pages and the display content.|
|2.3||High||Test the "Browse" use case to browse through titles based on the personalized information.|
|2.4||High||Test the "Search" use case to search for particular titles.|
|2.5||High||Test the "Order the Hard Copy" use case for ordering a hard copy of a book from the publisher.|
|2.6||High||Test the "Order the Soft Copy" use case for ordering a soft copy of a book from the publisher.|
Table 9.10 summarizes the expected results for execution of each test case from Table 9.19 (listed in the "Appendix" section later in this chapter). You should note that Table 9.19 lists only one of the possible flow paths for the use case. If the use case involves any alternate flow paths, you should test all of the flow paths with different combinations of input data.
Table 9.10: Sample Expected Results for the Online Bookstore Sample Application
|Serial number||Scenario to be tested|
|2.1||Registering with the Web Site||The user should be able to register without errors and the profile should be stored in the database, defined by the connection string that has been stored as administrative configuration information in the Oracle data store.|
|2.2||Personalization||The user should be able to personalize the Web page style and content to be displayed, and the personalization information should be stored in the Oracle data store.|
When the user logs on the next time, the profile information should be retrieved, and customized content should be displayed on the Web page. The information is retrieved from the database based on the connection string that is part of the administrative configuration data.
The warning, information, and exception events should be logged according to the settings in the configuration information that is retrieved through the CMAB from the Oracle data store.
|2.5||Order the Hard Copy||The user should be able to order a hard copy from the publisher. |
Profile information should be retrieved from the database based on the connection string retrieved as part of the administrative configuration data through the CMAB from the Oracle data store.
The appropriate publisher Web services should be contacted based on the URLs retrieved from the configuration data stored through the CMAB in the Oracle data store.
The warning, information, and audit and exception events should be logged according to the settings in the configuration information that is retrieved through the CMAB from the Oracle data store.
The payment authorization should be done according to the settings in the configuration information that is retrieved through the CMAB from the Oracle data store.
Event notification should be published according to the settings in the configuration information that is retrieved through the CMAB from the Oracle data store.
Step 3: Perform Load Testing
Load testing of the integrated application is required to analyze its behavior at various load levels. Load testing helps to verify that the application meets the performance objectives required for the application after integrating the application block. Because of the integration, there may be bottlenecks in the applicationsuch as resource contention with other modulesthat could reduce application performance. Load testing helps to identify such bottlenecks that usually surface only when the application is subjected to a load of concurrent users.
The test cases are the same test cases that you would execute for the application, whether or not you are integrating an application block.
The steps to perform load testing follow the load testing process described in Chapter 16, "Testing .NET Application Performance" of Improving .NET Application Performance and Scalability on MSDN at:
The steps to perform load testing are the following:
- Identify the test scenarios.
- Identify the workload.
- Identify the metrics.
- Create the test cases.
- Simulate the load.
- Analyze the results.
The next sections describe each of these steps as they apply to load testing.
Identify the Test Scenarios
The inputs for this step can be obtained from the following sources:
- Functional specifications or usage scenarios of the online bookstore sample application and application block
- Requirements for the sample application and application block
- Performance objectives for the sample application
- Performance model
The selected usage scenarios of the application are ones that integrate and use application block.
Consider the online bookstore sample application. The CMAB has been integrated to manage the configuration information for this application. The sample usage scenarios are listed in Table 9.19 in the "Appendix" section later in this chapter.
Identify the Workload
The second step in the process of load testing is the identification of workload. The input for the workload identification is taken from the performance model, marketing data, requirements, and the test scenarios identified in the preceding step. The key components of a workload model for the application are as follows:
- The maximum expected number of simultaneous users logged on to the application. These users have active connections to the Web site; however, all of them may not be sending requests at any given time. The excepted load for the online bookstore sample application is 1000 simultaneous users.
- Various user profiles. In a real-world environment, a user might log on, personalize a Web page, search or browse for items, and then purchase the books. The important point is that each user does multiple operations. Therefore, it is important to have various user profiles and to find out the appropriate distribution of the maximum load among these user profiles. Each user profile may have one or more dominant use cases that the user of that profile most frequently requests, as compared to others.
In the case of the online bookstore sample application, the typical operations are as follows:
- Log on
- Buy soft copy
- Buy hard copy
Table 9.11 shows the operations that constitute a Browse user profile. Ideally you would study the existing profiles of similar existing applications or conduct market research to develop suitable user profiles for your application.
Table 9.11: Sample Browse User Profile for Online Bookstore Sample Application
|Operation||Number of times performed in a single session|
|Log on use case||1|
|Browse use case||6|
|Search use case||2|
Profiles for the sample application are the following:
- Register profile
- Personalize profile
- Browse profile
- Search profile
- Buy Soft Copy profile
- Buy Hard Copy profile
At any given time, a Web application has users from various profiles logging on. Therefore, the workload for an application consists of a user profile mix, with each profile constituting a percentage of the total users logged on. Table 9.12 shows the user profile mix for the sample application.
Table 9.12: Sample User Profile Mix for Online Bookstore Sample Application
|User profile||Percentage||Simultaneous users|
|Buy Soft Copy||15||150|
|Buy Hard Copy||10||100|
- Think time between requests. Think time is the time that a user spends between two consecutive requests. You should estimate the think time based on the complexity of the Web page. The think time needs to be higher if the user needs to fill in personal details, as is the case with "Registering with the Web site" use case. Based on the complexity, calculate the average think time for each of the requested pages. The think time can be anywhere from 5 seconds to more than a minute.
Identify the Metrics
The third step in the process of load testing is the identification of metrics. Metrics are used to determine quantitatively whether the performance goals specified in the requirements are being met and are also used to identify any bottlenecks affecting the performance and scalability of the application. Metrics related to performance goals may be available from System Monitor, or you may need to instrument the code to capture custom metrics. For example, the online bookstore sample application has performance goals for the following, which require custom instrumentation of the code:
- Average number of read operations from the Oracle data store per second
- Average number of write operations to the Oracle data store per second
- Average time for a title search to return results
- Average time for a user to log on
- Average time to submit the personalized information
Metrics in the initial iterations generally give a coarse-grained picture of the overall behavior of application. During subsequent iterations, you can add more metrics to focus on a specific area for a fine-grained picture. For example, if you see an increase in private bytes during testing, you can add counters related to various managed heaps.
In the case of the online bookstore sample application, where hundreds of concurrent users store and read personalized information, browse and search product catalogs, and so on, you need to monitor the disk-related counters because there is a possibility of the disk becoming a bottleneck on the server hosting the Oracle data store.
You should monitor a set of counters (shown in Table 9.13) related to resource utilization on all of the servers. These counters help you to analyze the overall resource utilization of the servers and to determine whether any of the resourcessuch as processor, memory, disk, or network I/Oare a bottleneck.
Table 9.13: Base Set of Metrics for All Servers in the Deployment
|Processor||% Processor Time||_Total|
|Processor||% Interrupt Time||_ Total|
|Processor||% Privileged Time||_ Total|
|System||Processor Queue Length||Not applicable|
|System||Context Switches/sec||Not applicable|
|Memory||Available Mbytes||Not applicable|
|Memory||Cache Faults/sec||Not applicable|
|Physical Disk||Avg. Disk Queue Length||(Disk instance)|
|Physical Disk||Avg. Disk Read Queue Length||(Disk instance)|
|Physical Disk||Avg. Disk sec/Read||(Disk instance)|
|Physical Disk||Avg Disk sec/Write||(Disk instance)|
|Server||Pool Nonpaged Failures||Not applicable|
|4Process||Working Set||(Monitored process)|
|Process||Private Bytes||(Monitored process)|
|Process||Handle Count||(Monitored process)|
Table 9.14 lists the counters that need to be captured during the first iteration of the load testing. You can add more counters based on the analysis in subsequent iterations.
Table 9.14: Metrics to Be Monitored on the Web Tier
|ASP.NET||Requests Current||Not applicable|
|ASP.NET||Requests Queued||Not applicable|
|ASP.NET||Requests Reject||Not applicable|
|ASP.NET||Request Execution Time||Not applicable|
|ASP.NET||Request Wait Time||Not applicable|
|ASP.NET Applications||Requests/Sec||Your virtual directory|
|ASP.NET Applications||Requests Executing||Your virtual directory|
|ASP.NET Applications||Requests in Application Queue||Your virtual directory|
|ASP.NET Applications||Requests Timed Out||Your virtual directory|
|ASP.NET Applications||Cache Total Hit Ratio||Your virtual directory|
|ASP.NET Applications||Cache API Hit Ratio||Your virtual directory|
|ASP.NET Applications||Output Cache Hit Ratio||Your virtual directory|
|ASP.NET Applications||Errors Total/sec||Your virtual directory|
|ASP.NET Applications||Pipeline Instance Count||Your virtual directory|
|.NET CLR Memory||% Time in GC||Aspnet_wp|
|.NET CLR Memory||# Bytes in All Heaps||Aspnet_wp|
|.NET CLR Memory||# of Pinned Objects||Aspnet_wp|
|.NET CLR Memory||Large Object Heap Size||Aspnet_wp|
|.NET CLR Exceptions||# of Exceps Thrown/sec||Aspnet_wp|
|.NET CLR LocksAndThreads||Contention Rate/sec||Aspnet_wp|
|.NET CLR LocksAndThreads||Current Queue Length||Aspnet_wp|
|.NET CLR Data||SqlClient: Current # connection pools||Not applicable|
|.NET CLR Data||SqlClient: Current # pooled connections||Not applicable|
|Web Service||ISAPI Extension Requests/sec||Your Web site|
Create the Test Cases
The fourth step in the process of load testing the integrated application is to create the test cases. The test cases are created based on the scenarios identified in Step 1 and workload profiles identified in Step 2. You can create test cases for incremental load testing by increasing the load for a given workload profile until the metrics hit their performance threshold, or you can create a test case for analyzing performance for a specific load of users. Table 9.15 shows a test case with a workload profile and a load of 1,000 simultaneous users.
Table 9.15: Sample Test Case for the Online Bookstore Sample Application
|Workload profile||User profile||Percentage||Simultaneous users|
|Buy Soft Copy||15||150|
|Buy Hard Copy||10||100|
|Test duration||60 minutes|
|Think time||Random think time of 110 seconds|
|Throughput||100 requests per second (ASP.NET\Requests/sec performance counter)|
|Memory\Available Mbytes||25 percent of total RAM|
|Average response time||<3 seconds (on 100 megabits per second [Mbps] LAN)|
Simulate the Load
You can simulate the load by using load simulation tools such as Microsoft Application Center Test (ACT). You should make sure the load generator does not stress the resources (such as processor or memory) on the client computer.
Analyze the Results
The final step in load testing the application is analyzing results. The metrics are analyzed for any bottlenecks that are hurting application performance and to determine whether the application performance is moving toward or away from the performance objectives.
If you encounter any bottlenecks, you need to identify the cause by either conducting a code review of the module or profiling the code using tools from the debugging toolkit for the Microsoft® Windows® operating system.
If the application block is the cause of the bottleneck, you need to unit test and fix the issues in the application block, and then continue with integration testing.
Step 4: Perform Stress Testing
The integrated application needs to be stress tested to evaluate application behavior at loads greater than the peak operating capacity while the resources to process the load are being denied. This type of testing reveals issues that arise only under abnormal loads, such as synchronization issues, loss of data during network congestion, memory leaks, and so on.
To stress test the integrated application block, you need to identify scenarios, workload profiles, metrics, and test cases. You should use input from load testing, such as the test scenarios, peak operating capacities, workload allocations, and so on, as well as code review results to formulate the test cases for stress testing.
The steps for stress testing are as follows:
- Identify the key scenarios.
- Identify the workloads.
- Identify the metrics.
- Create the test cases.
- Simulate the load.
- Analyze the results.
The next sections describe each of these steps in more detail.
Identify the Key Scenarios
The first step in stress testing is to identify the scenarios that are critical to performance of the overall application. This includes operations such as locking, synchronized access to a resource, or extensive disk or network I/O. You can identify these scenarios based on input from load testing. Usually, such scenarios are resource intensive or occur frequently and affect overall performance directly or indirectly.
For example, in the online bookstore sample application, each usage scenario requires different configuration information. If caching is not enabled for the CMAB, the CMAB must repeatedly access the Oracle data store to read and write configuration information. When the number of users increases to very high loads, the number of CMAB requests to the Oracle database also increases significantly. This increase can lead to excessive disk I/O at the database server, which in turn leads to queue buildup at the database server and an increase in response time. The requests on the Web server continue to wait for a response from the database server, and hold on to the threads from the thread pool. This can cause the requests to time out, and the user may receive "time out" or "server too busy" errors.
Another possible scenario deals with enabling caching for the CMAB, and then caching a large amount of data on the Web server. This significantly increases memory utilization on the Web server and, in worst case, may cause the worker process to recycle.
Table 9.16 lists other stress testing scenarios from the sample application.
Table 9.16: Sample Stress Testing Scenarios for the Online Bookstore Sample Application
|Scenario 1||Conduct stress testing of various use cases of the online bookstore sample application|
|1.1||High||Test the Personalize profile for a number of users above the peak operating capacity identified in load testing.|
|1.2||High||Test the Buy Soft Copy profile for a number of users beyond the peak operating capacity.|
|1.3||High||Test the Buy Hard Copy profile for a number of users beyond the peak operating capacity, with a large number of users ordering hard copies of the book from the publishers.|
Identify the Workloads
The second step in stress testing is to identify workload profiles based on the scenarios identified in the preceding step. You can use the same workload profiles from load testing but increase the load for the dominant use case significantly or stress test the use case in isolation. The input for the workload identification includes the peak operating capacities taken from load testing for a given workload profile.
The workload is usually increased from a value beyond the maximum operating capacity. For example, in the case of the test scenario, if the maximum operating capacity is 1,000 users, then the workload can be varied between 1,100 to 2,000 or more users.
Identify the Metrics
Metrics are specific to the test scenarios, and you should select them based on the seriousness and likelihood of potential problems.
For example, in the case of stress testing with a workload and storing a large amount of information in the application block cache, you should monitor the memory-related counters for the system, the ASP.NET worker process, and the base set of counters listed in Table 9.13. Table 9.17 lists the performance counters you would most likely want to monitor for this scenario.
Table 9.17: Metrics to Be Captured for Stress Testing the Online Bookstore Sample Application
|ASP.NET Applications||Requests/Sec||<Virtual directory>|
|ASP.NET Applications||Requests Executing||Not applicable|
|ASP.NET Applications||Requests Timed Out||Not applicable|
|ASP.NET Applications||Requests in Application Queue||<Virtual directory>|
|ASP.NET||Request Execution Time||Not applicable|
|ASP.NET||Requests Queued||Not applicable|
|ASP.NET||Requests Execution Time||Not applicable|
|ASP.NET||Requests Rejected||Not applicable|
|.NET CLR Memory||# Bytes in All Heaps||Not applicable|
|.NET CLR Memory||# Gen 0 Collections||Not applicable|
|.NET CLR Memory||# Gen 1 Collections||Not applicable|
|.NET CLR Memory||# Gen 2 Collections||Not applicable|
|.NET CLR Memory||% Time in GC||Not applicable|
|.NET CLR Memory||Large Object Heap Size||Not applicable|
Create the Test Cases
Document the test cases using the workload patterns and scenarios that you identified based on the results of the previous steps.
Table 9.18 shows the results for the sample application.
Table 9.18: Test Case Results for the Online Bookstore Sample Application
|Scenario||A user orders a hard copy of a book from the publisher|
|Number of users||2000 simultaneous users|
|Test duration||1 hour|
|Think time||Random think time is 110 seconds|
|Expected results||The ASP.NET worker process should not be recycled.|
Throughput should not fall below 30 requests per second.
(ASP.NET\Requests/sec performance counter)
Response Time for the usage scenario should not exceed 7 seconds (on a 100 Mbps per second LAN)
Server busy errors should not exceed 20 percent of the total response.
Memory utilization should not be more than 60 percent of available memory (Memory\Private Bytes)
Simulate the Load
Simulate the load as specified in the test cases by using load simulation tools such as ACT.
Analyze the Results
The final step in stress testing the application is to analyze the results. If you identify any potential bottlenecks caused by the application block rather than the application itself, you need to conduct white box testing of the application block by profiling or debugging the code using various tools, such as debugging tool kit.
For more information about analyzing the results of stress testing, see the following resources:
- For more information about tools and techniques for debugging, see "Production Debugging for .NET Framework Applications" at:
- For more information about debugging deadlocks, see "Scenario: Contention or Deadlock Symptoms" in "Production Debugging for .NET Framework Applications" at:
- For more information about deadlocks, see Microsoft Knowledge Base article 317723, "INFO: What Are Race Conditions and Deadlocks?" at:
Step 5: Perform Globalization Testing
One requirement of the online bookstore sample application is that the application should be easily localizable for Japanese and German locales. This means that the application should be able to accept, process, store, and display Japanese and German characters. The resulting localized application is expected to meet the requirements of the functional specifications and the performance objectives of the application.
The test cases and test scenarios executed as part of functional testing also apply for the functionality testing in globalization. The test cases must be tested for both German and Japanese locales and cultures in the case of the sample application.
You will need to set up the test environment, which may include one or both of the following:
- You may need a target version of the operating system with the appropriate language group installed. The targeted language groups in the case of the sample application are Japanese and German.
- You may need a localized build target of the operating system, for example, the Windows operating system with Japanese as the UI language.
After the environment is set up, you need to execute the application use cases to ensure that the localized application functions as expected. Even though the application block has been tested for globalization in the unit testing phase, it is important to verify that the use cases that integrate the application block work correctly.
For more information about globalization testing, see the following resources:
- For more information about step-by-step globalization testing for integrated components, see "Globalization Step-by-Step" on the Microsoft Global Development and Computing Portal at:
- For more information about best practices for globalization, see "Best Practices for Developing World-Ready Applications" in the .NET Framework Developer's Guide on MSDN at:
Step 6: Perform Security Testing
You should test the security of the sample application after it is integrated with the application block. This testing should focus on identifying any new vulnerabilities that were introduced as a result of the integration. This testing can be done at two levels:
- Black box integration testing. The focus of this testing should be on the application block integration. You should test for unauthorized access to ensure that the application is secure after integration. This type of testing is done by simulating the production environment, and it attempts to find issues that could arise at run time. The test cases that were created during the application block black box testing phase can be used here by modifying them to use the changed interface. You can also use any test cases developed earlier for the application to ensure that the application is secure.
- White box integration testing. The application threat model should be updated to include application block integration. (It is assumed that the sample application already has a threat model in place.) The new threat model should do the following:
- Update the application architecture diagram with the application block architecture.
- Identify any new assets, privilege requirements, and resources accessed by the application block.
- Based on additional information collected earlier, identify any new threats and update the threat model.
- Prioritize the threats and suggest countermeasures.
Any threats you identify during security testing can be a result of one or both of the following:
- The application block design and/or code did not follow security best practices, such as not supporting encryption if the application block is used for handling sensitive data or not using the best encryption algorithm available to encrypt sensitive data.
- The application block failed to consider specific requirements of the application, such as the application block depending on a service that makes the application vulnerable.
You should also review the integration-related code to ensure that it follows security-related best practices.
For a detailed description of threats and countermeasures and the process of threat modeling, see Chapter 3, "Threat Modeling" in Improving Web Application Security: Threats and Countermeasures on MSDN at:
This chapter presented the processes for unit testing the customizations made to the application block and how to test the application block after you integrate it with your application. The process of integration testing assumes that the application block has been rigorously unit tested and meets all requirements of the application, and that the developer has done a careful analysis before deciding to use the application block.
Table 9.19: Execution Steps for the Online Bookstore Sample Application Uses Cases
|Test scenario||Steps to follow|
|Registering with the Web site||Fill out the logon form or the user profile.|
Submit the user profile.
|Personalizing the Web pages and content to be displayed||Log on.|
Modify or upload the Web page templates.
Modify the information to be displayed.
Modify the number of titles to be listed per page.
Choose whether the titles displayed should be categorized based on the publisher or the subject matter of the books.
Choose the order in which the titles are listed, either in alphabetical order or in the chronological order.
Choose the publishers whose titles should be listed.
Submit the personalized information.
Choose the method of payment and the bank.
Enter the credit card and or bank account details.
|Browsing through the application||Log on.|
Browse the advertisements and new titles that are listed based on the history of a specific user.
Read the abstract and user preface for each new title.
Browse the latest books from each publisher.
|Searching for a title||Log on.|
Search for a book.
Read the abstract and user preface for the selected book.
|Reserving a book||Log on.|
Search for a book.
Select the distributor.
Reserve the book.
|Buying the soft copy of the book||Log on.|
Browse through the latest offerings from a publisher.
Select a book.
Choose the soft copy format of the book.
Certify and submit the payment details.
Download the soft copy.
|Ordering the hard copy of the book from the publisher||Log on.|
Browse through the latest offerings from a publisher.
Select a book.
Choose the hard copy option of the book.
Choose the option to have the publisher send the book.
Certify and submit the payment details.