Export (0) Print
Expand All
7 out of 9 rated this helpful - Rate this topic

Testing Process for Application Blocks

Retired Content

This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

 

patterns & practices Developer Center

Microsoft Corporation

January 2005

Summary: Covering the overall description of the various test categories and techniques used for testing an application block.

Contents

Objective

Overview

Testing Process

Summary

Appendix

Objective

  • Learn the process for testing application blocks.

Overview

When testing application blocks, you follow the same basic steps regardless of the methodologies you use to develop and test the block. The purpose of these steps is to test the application block from different perspectives and to ensure that the application block conforms to all of the requirements and functional specifications.

This chapter describes the process you follow to test application blocks. If required, you can customize the process to suit your development strategy. Before testing the application blocks, you should conduct a risk analysis of the features and the importance of the functionality that the application blocks provide. The risk analysis determines the amount of testing effort you expend for each step.

Testing Process

The process for testing application blocks is outlined below. The steps listed should be performed regardless of the testing methodology that is used for the blocks.

Input

The following input is required to test an application block:

  • Functional specifications
  • Requirements
  • Performance objectives
  • Deployment scenarios

Steps

Figure 3.1 shows the steps in the testing process for application blocks.

Ff649517.f03mtf01(en-us,PandP.10).gif

Figure 3.1. The testing process for application blocks

  1. Create test plans. Create test plan documentation with a prioritized list of test cases and execution details.
  2. Review the design. Review the design to ensure that it addresses all the deployment scenarios and the requirements. Also ensure adherence to guidelines for performance, security, globalization, maintainability, and so on.
  3. Review the implementation. Review the code to ensure that various best practices are followed.
  4. Perform black box testing. Test the code from an end-user perspective by executing all the end-user scenarios.
  5. Perform white box testing. Analyze the code for failure scenarios and simulate them by passing invalid data using custom test harnesses.

Step 1: Create Test Plans

The test plans document the test cases you will use to test the application block. The test cases cover all aspects of testing, including design review, code review, profiling, deployment testing, and load testing. The test plans help to ensure that you test all of the features and usage scenarios of the application block.

The test plan documentation consists of two documents:

  • Detailed test plan (DTP) document. The detailed test plan document lists test cases in a prioritized order (High, Medium, and Low). The test case descriptions in this document briefly summarize the usage scenarios and features to be tested. For each test case, you assign a priority level based on the case's importance and its overall impact on the goal of meeting the desired objectives and requirements.
  • Detailed test case (DTC) document. The detailed test case document maps to the detailed test plan document. This document describes the steps that the user must perform to execute each test case that is listed in the DTP document. It also lists the data that is required for the test and describes the expected results of the test.

For examples of DTP and DTC documents, see the "Appendix" section later in this chapter. You can use these examples as templates when you create the test plans for your application block.

You must update the DTP and DTC documents throughout the development life cycle. For example, you must update these documents if functional specifications or requirements change, or if you have additional input. You must also update them if you add test cases or if you modify the priority of the existing test cases later in the development cycle to account for additional usage scenarios or functional testing.

Step 2: Review the Design

In this step, you execute the test cases from the DTP that are related to reviewing the design of the application block. The design review is very important from the testing point of view, because you can avoid most of the potential pitfalls (for example, those related to performance or security) in the early stages of a project by thoroughly reviewing the design of the application block. In the later stages, considerable rework may be required in the code to make any design changes.

The design review helps to ensure the following:

  • The design meets all of the functional specifications and requirements for the application block.
  • The design makes appropriate tradeoffs given the deployment scenarios for the application block. Examples of deployment targets that may have special requirements are mobile devices, the DMZ (also known as demilitarized zone or screened subnet) of a Web farm, and tablet PCs.
  • The design ensures that the application block meets all performance objectives.
  • The design addresses all the security threats possible in various deployment scenarios for the application block.
  • The design adheres to best practices and principles related to coupling and cohesion, concurrency, communication, class design, exception management, resource management, caching, and so on, so that developers can easily extend and customize the application block.
  • The design adheres to best practices for globalization and localization.

The design review also helps in identifying the scenarios that must be tested for one or more of the following:

  • Possible security attacks
  • Performance optimizations
  • Profiling to ensure that there are no memory leaks
Note   You must update the DTP and DTC to document these possible scenarios.

More Information

For more information about how to conduct a design review for the application block, see Chapter 4, "Design Review for Application Blocks."

Step 3: Review the Implementation

In this step, you execute the test cases from the DTP that are related to reviewing the implementation (code) of the application block. This is an important step in unit testing the application block. By conducting an extensive code review, you can detect errors early in the life cycle. If you do not detect errors until the white box testing stage, correcting them may require significant and costly changes to the system, which may lead to a cascading effect of code changes between various modules.

The implementation of the application block is reviewed for the following:

  • The code adheres to the design that was laid out in Step 1.
  • Naming guidelines for code elements (class names, variables, method names, and assemblies, and so on) are followed.
  • The code has comments at appropriate places to help customers understand the application block so they can easily customize it.
  • The code follows best practices related to the following:
    • Performance and scalability
    • Writing secure code
    • Ensuring that relevant design patterns are being followed and looking for any possible anti-patterns in the code
    • Exception management
    • Resource management (memory allocation, connection pooling, object pooling, and so on)
    • Globalization and localization
  • The block does not contain redundant or commented code that is never called.

The code review also helps in identifying the possible scenarios that must tested for one or more of the following:

  • Boundary level conditions
  • Special inputs
  • Possible security attacks
  • Performance optimizations
  • Profiling to ensure that there are no memory leaks
  • Thread safety and deadlock issues
Note   You must update the DTP and DTC to document these possible scenarios.

More Information

For more information about implementation review of the application blocks, see Chapter 5, "Code Review for Application Blocks."

Step 4: Perform Black Box Testing

In this step, you execute the test cases in the DTP that are related to black box testing of the application blocks. Black box testing is testing without knowledge of the internal workings of the system that is being tested. This type of testing simulates end-user experience.

Black box testing helps to ensure that the application block:

  • Meets all of the objectives listed in the requirements document.
  • Covers all of the functionalities that are specified in the functional specifications.
  • Can gracefully handle all of the expected and exceptional usage scenarios. The error messages displayed are meaningful and help the user in diagnosing the problem.

You may need to develop one or more of the following to test the functionality of the application blocks:

  • Test suites, such as NUnit, to test the API of the application block for various types of input.
  • Dummy Windows forms or Web Forms applications that integrate the application blocks and are deployed in simulated target deployments.
  • Automated scripts that test the APIs of the application blocks for various types of input.

The black box testing assumes that the tester has no knowledge of code and is intended to simulate the end-user experience. You can use sample applications to integrate and test the application block for black box testing. This approach tests all possible combinations of end-user actions. The test planning for black box testing can begin as soon as the requirements and the functional specifications are available. Black box testing includes the following:

  • Testing all of the external interfaces for all possible usage scenarios. This means testing all of the external interfaces (such as public classes rather than the private or internal classes) that end users can integrate with their applications. Testing the external interfaces involves the following:
    • Ensuring that the interfaces meet the functional specifications and address all of the requirements. This type of testing ensures that the application block provides interfaces for meeting all of the requirements as stated in the functional specifications. This type of testing requires that you develop test suites. You must test for all of the ways in which clients of the application block can call the APIs. The usage scenarios include both the expected process flows and random input.
    • Testing for various types of input. This type of testing ensures that the interfaces return the output as expected and are robust enough to handle invalid data and exceptional conditions gracefully. The input data can be randomly generated either within a specified range expected by the application block, outside the specified range, or at the boundary of the range (known as boundary testing). Testing with data outside the specified range ensures that the application block is robust and can handle invalid data, and that the error messages generated are meaningful for the end user. Boundary testing ensures that the highest and lowest permitted values for input produce expected output.
  • Performance testing. You execute performance-related test cases from the DTP in a simulated environment that is close to the real-world deployment. Performance testing verifies that the application block can perform under expected normal and peak load conditions, and that it can scale sufficiently to handle increased capacity. There are two main aspects of performance testing: load testing and stress testing. Each aspect has different end goals. You must plan and execute test cases for both aspects as follows:
    • Load testing. Use load testing to verify the application block behavior under normal and peak load conditions. Load testing enables you to verify that the application block can meet the performance objectives and does not overshoot the allocated budget for resource utilization such as memory, processor, and network I/O. It also enables you to measure response times and throughput rates for the application that is using the application block.

      Load testing also helps you to identify the overhead (if any, in terms of resource cost such as processor, memory, disk i/o or network i/o) of using the application block to achieve a desired functionality by testing applications with and without the application block to achieve the same result.

    • Stress testing. Use stress testing to evaluate the application block's behavior when it is pushed beyond peak load conditions. The goal of stress testing is to identify errors that occur only under high load conditions—for example, synchronization issues, race conditions, and memory leaks.

    The analysis from performance tests can serve as input for the implementation review and white box testing. If you detect a problem, you may need to review the code of the module you suspect to isolate the cause. (For example, if you have a problem of increased response wait time, a code review may reveal that a coarse-grained lock is causing the problem). During the white box testing stage, you can profile the suspected code path and optimize it.

  • Security testing. Security testing from the black box perspective is done by simulating the target deployment environment and performing tests that simulate the actions an attacker can take in a real-world production environment.

You identify vulnerabilities in the application block (and hence expose threats) by attempting to simulate all possible types of input. The goal is to expose (at run time) any threats that may exist in the code because of poor design or poor coding practices. The scope of security testing for an application block is different from that for a real-world application and depends on the type of functionality that the application block provides. You can execute tests to:

  • Check input validation techniques, if the application block has functionality that validates input data. You can provide various types of input with the objective of breaking the validation technique.
  • Break the encryption and access sensitive data, if the application block handles sensitive data and uses encryption.
  • Check for buffer overflows.
  • Check for cross-site scripting errors.
  • Validate authorization and authentication mechanisms, if the application block provides this functionality.
  • Globalization testing. The design and implementation have already been reviewed for adherence to best practices for globalization. For black box testing, you execute the globalization testing–related test cases from the DTP to ensure that the application block can handle international support and is ready to be used from various locales around the world.

    The goal of globalization testing is to detect problems in application design that could interfere with globalization. Globalization testing ensures that the code can handle all international support and that it supports any of the culture/locale settings without breaking functionality that would cause either data loss or display problems.

    To perform globalization testing, you must install multiple language groups and ensure that the culture/locale is not your local culture/locale. Executing test cases in both Japanese and German environments, for example, can cover most globalization issues.

More Information

For more information, see the following resources:

Step 5: Perform White Box Testing

In this step, you execute the test cases in the DTP that are related to white box testing (also known as glass box, clear box, or open box testing) of the application blocks. White box testing derives the test cases and the related test data by analyzing the internal workings and program logic of the system.

White box testing assumes that the tester can look at the code for the application block and create test cases that test for any potential failure scenarios. This exercise is done to some extent in Step 3, where you can identify the potential scenarios for additional testing.

To determine what input data to use to test various APIs and what special code paths you must test, you analyze the source code for the application block. Therefore, you must update the test plans before you begin the white box testing.

The failure of a white box test may result in a change that requires you to repeat all of the black box testing and to redetermine the white box paths.

White box testing includes the following:

  • Profile the application block. Profiling is the activity that allows you to monitor the behavior of a particular code path at run time when the code is actually being executed. You can profile your application block for one or more of the following:
    • Code coverage. Profiling for code coverage with the help of a tool ensures that your application does not contain redundant or commented code that is never called for any execution path.
    • Memory allocation pattern. Profiling the memory allocation helps you in studying direct and indirect memory allocations while executing a code path and detecting and analyzing memory leaks and other problems. For example, you can determine which generations (Gen 0, Gen 1, Gen2) contain the maximum number of objects, or whether any side effect allocations are taking place in a loop and are increasing the memory utilization significantly. (For example, using a string class for concatenation in a loop can cause many additional string object allocations.) Doing so can help you to optimize the memory utilization of the application block. You can use various profiling tools, such as CLR Profiler and Intel VTune, to analyze the memory allocation.
    • Contention and deadlock issues. You can analyze the application block for deadlocks by using tools such as WinDbg from the Microsoft Windows debugging toolkit.
    • Time taken to execute a code path. You can profile the time taken by the scenarios for which performance is critical. This profiling may require custom instrumentation of the code path, which may be distributed across separate computers. Various third-party tools are also available to help you measure the time taken to execute a particular scenario.

      If the application block needs to be integrated with a Web application, you may also want to monitor Request Execution Time (ASP.NET\Request Execution Time performance counter) for a test suite that uses the application block. The overhead of the suite should be minimal, and most of the processing time should be spent within the application block.

    If, during profiling, you find that a particular condition must be tested under load (such as potential deadlock issues or inefficient memory cleanup) you can provide input on the type of metrics that should be used while load testing the application block:

  • Test internal subroutines. You must test the various internal functions to ensure that they process the right data without any loss or inconsistency and return the expected output.
  • Test various loops and conditional statements. You should test various types of loops (for example, simple, concatenated, and nested loops) and conditional statements (for example, relational expressions, simple conditions, compound conditions, and Boolean expressions) in the code components for accuracy.
  • Test various scenarios identified in Step 3. You should test the scenarios you identified in the implementation review (Step 3). Analyze the code, and then submit the input to identify weaknesses in the code.
  • Test for security. If your source code review reveals that a particular code access security configuration cannot restrict undesirable access in the targeted deployment environment, you should test the particular feature consuming the code path. Analyze the code, and then simulate the deployment environment to identify any scenarios in which the application block may expose sensitive information to attackers.

More Information

For more information on white box and black box testing of the application block, see Chapter 6, "Black Box and White Box Testing for Application Blocks."

For a discussion of code coverage analysis, see "Perform Code Coverage Analysis with .NET to Ensure Thorough Testing" in MSDN Magazine at http://msdn.microsoft.com/msdnmag/issues/04/04/CodeCoverageAnalysis/default.aspx.

Summary

The chapter presents a process for testing an application block. The concepts that are outlined in the process apply regardless of the software development life cycle approach followed. The amount of effort and the number of iterations depend on the risk analysis of the overall significance of the application block at the macro level and each of the features at the micro level.

Appendix

Sample Test Case from a Detailed Test Plan Document

Scenario 1Developing and executing NUnit test suite to test the various public classes available for the application block.
PriorityHigh
Comments 
1.1HighDeveloping and executing NUnit test suite to test the reading and writing to the registry, SQL Server, and XML file storage media by using the Configuration Management Application Block.

Sample Test Case from a Detailed Test Case Document

Test casePriorityCondition to be testedExecution detailsData requiredExpected resultsActual resultTest OK

(Y/N)

1.1HighDeveloping and executing NUnit test suite to test reading and writing to the registry, SQL Server, and XML file storage media by using the Configuration Management Application Block.1. Create an assembly, ConfigurationManagement.UnitTests.

2. Create the following .cs files in the above assembly:
(Note that the static public methods of ConfigurationManager will be used for the read and write operations.)
XmlFileConfigurationManagement.cs
In this class, implement methods to read and write a custom object to an XML file for all combinations of the provider and cache settings for encryption and signing.
RegistryConfigurationManagement.cs
In this class, implement methods to read and write a custom object to the registry for all combinations of the provider and cache settings for encryption and signing.

SqlServerConfigurationManagement.cs
In this class, implement methods to read and write a custom object to a SQL Server database for all combinations of the provider and cache settings for encryption and signing.

Performance.cs
In this class, write methods to:
--Test the response time per read or write request for a large number of requests.
--Test the total number of requests (for read and write) served in a second.

After writing the methods, test the above two cases for all of the persistent media, XmlFileStorage, RegistryStorage, and SqlServerStorage.

Stress.cs
In this class, write methods to:
-Test multiple read and write from multiple threads,
-Read and write bulky objects

After writing the methods, test the above two cases for all of the media, XmlFileStorage, RegistryStorage and SqlServerStorage

3. Compile the above assembly.
4. Run NUnit on this assembly.
Configuration Management Application Block, NUnitAll NUnit test cases should run successfully for ConfigurationManagement. UnitTests.dll.  

Start | Previous | Next

patterns & practices Developer Center

Retired Content

This content is outdated and is no longer being maintained. It is provided as a courtesy for individuals who are still using these technologies. This page may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist.

Show:
© 2014 Microsoft. All rights reserved.