Testing for Reliability
Testing for reliability is about exercising an application so that failures are discovered and removed before the system is deployed. Because the different combinations of alternate pathways through an application are high, it is unlikely that you can find all potential failures in a complex application. However, you can test the most likely scenarios under normal usage conditions and validate that the application provides the expected service. As time permits, you can apply more complicated tests to reveal subtler defects.
This section contains a selection of testing concepts and recommendations that are especially relevant to creating reliable applications. Additional testing information relevant to high-availability applications is at Testing for Availability. For more complete information on the general subject of testing, search for "testing" in the MSDN Online Library (http://search.microsoft.com/us/dev/default.asp).
Use Component Stress Testing
Stress testing is about simulating large workloads to see how the application performs under peak usage conditions. With component stress testing you isolate the constituent components and services, figure out what navigational, functional, and interface methods they expose, and create a test front end that calls those methods. For those methods that go to a database server or some other component, you create a back end that provides dummy data in a format it expects. The test apparatus inserts dummy data over and over while observing the results.
The idea here is to stress each component in isolation far beyond what the normal application would experience. For example, use a 1 – 10,000,000 loop just as fast as possible and see if there are discoverable problems. Testing each DLL by itself helps identify gross failures with the component.
For distributed Web applications, Microsoft provides a Web Application Stress Tool. For more information, see Microsoft Web Application Stress Tool (http://homer.rte.microsoft.com). If you purchased Visual Studio .NET Enterprise edition, another tool called Application Center Test is provided as an introductory preview of some of the technology in Application Center 2000.
Use Integration Stress Testing
After stressing each individual component, you should stress the entire application with all of its components and supporting services. Integration stress testing is largely concerned about interactions with other services, processes, and data structures — from both internal components and other external application services.
Integration testing starts with just basic functional testing. You need to know the coded pathways and user scenarios, understand what the users are trying to do, and identify all the ways the user goes through your application.
Test scripts should exercise the application according to expected usage. For example, if your application displays a Web page where 99% of the customers will simply search the site and only 1% will actually buy, it makes sense to provide test scripts that stress the search and miscellaneous browse functions. Of course, the shopping cart should be tested too, but expected usage suggests that searching should comprise the majority of test cases.
Always extend the time of testing as much as your schedule and budget can allow. Instead of testing for a few days or a week, stretch the testing throughout a month, or through a quarterly cycle, or through a year-end cycle, and see how the application functions over a longer period of time.
Use Real-World Testing
Software that is reliable in isolation in a protected test environment may not be reliable in real-world deployment. While isolated testing is useful in the early reliability testing process, a real-world test environment ensures that concurrent applications do not interfere with each other. Such testing often discovers unexpected failure-causing interactions with other applications.
You need to ensure that the new application can run in the server space, in the final configuration, with the full experience of the expected customer event profile. The test plan should include running the new application in the final target environment or as close to the target environment as possible. This can usually be done with a partial replication or careful sharing of the final environment.
Use Random Destruction Testing
One of the easiest ways to test for reliability is by using random input. This type of testing endeavors to crash or hang the application by providing spurious, non-logical input. The input can be keyboard or mouse events, program message streams, Web pages, data caches, or any other input condition that can be forced into the application. You should use random destruction testing to test important error paths and expose bugs in your software. Such testing improves code quality by forcing failures so you can observe return error handling.
Random testing intentionally ignores any specification of program behavior. If the application breaks, it has failed the test. If the application does not break, it passes the test. The point here is that random testing can be highly automated because it is totally outside of concern with how the underlying application is supposed to work.
You'll need some kind of test harness to drive chaotic, high-stress, non-logical test events into the application's interface. Microsoft uses a tool called Injector that lets you inject faults into any API without requiring access to the source code. Injector can simulate resource failures, modify calling parameters, inject corrupt data, check for parameter validation boundaries, insert timing delays, and perform many other functions.
Obtain Windows 2000 Certification
The Microsoft certification program for applications helps validate reliable and manageable applications. This program is not for the timid: it includes rigorous testing that covers virtually every line item in the specifications and establishes public accountability. The application specification must be focused with clear testable functions, and all test plans, test tools, and test results are public. Certification provides high confidence that the application reliably delivers the required software services as identified in the specification.
Before getting your application formally tested, you should evaluate how well your application meets the specification and fix any discrepancies. There is a kit of tools containing test utilities, data, methods, and metrics that monitors compliance with Windows 2000 certification requirements. This test kit helps ensure that your application will meet the certification requirements for Windows 2000 when it is submitted for final testing. For more information on the test kit, see the Rational TestFoundation for Windows 2000 on the Rational home page (http://www.rational.com).
Certification is also important for 3rd-party applications that you intend to install on your system. Microsoft recommends that you ask software vendors to certify their applications and to adopt the Windows 2000 application guidelines for software programming.
To qualify for the "Certified for Windows 2000" logo, VeriTest, an independent testing laboratory, must test the application. VeriTest tests desktop, server, and client-server applications. For more information on Windows 2000 certification, see Testing Process (http://msdn.microsoft.com/certification/test.asp).