.gif)
Appendix A - Test Cases
Microsoft Corporation
February 2004
Summary: Discusses the testing approach for the offline application block.
Contents
Functional Testing
White Box Testing
Security Testing
Performance Testing
Integration Testing
Content Testing
Installation Testing
Appendix A describes the tests that were run against the Offline Application Block to ensure that it functions correctly. We include them as recommendations for the types of tests to consider when developing your own applications.
The tests cover seven areas:
- Functional testing, which ensures that the application meets the specified requirements
- White box testing, which tests small, specific areas of the code
- Security testing, which tests that the application and its data are protected and that privacy is protected and data properly encrypted
- Performance testing, which tests the application's processing and response times under various conditions
- Integration testing, which ensures that the application works with other systems and components
- Content testing, which verifies that the documentation is correct
- Installation testing, which verifies that the application installs correctly on the client machine
Each section of the document corresponds to one of these categories. For each category we provide two tables.
The first table includes:
- The scope, which is the area covered by this category of test
- When, in the development cycle, these tests should begin
- The tools used to perform the tests
- If automated tested is advisable for these types of tests
The second table is a checklist of sample tests that fall into the category.
Functional Testing
Functional testing ensures that the application meets the specified requirements.
Table 1: Functional Testing Definition
| Area | Definition |
|---|
| Scope | Tests cover the functionality of the smart client application to ensure that it meets specified requirements. They provide feedback on the validity and usability of the application to resolve any deviations between the requirements and the actual application. They exercise the graphical user interface (GUI), using both functional and scenario-based paths. |
| When to start | Tests should be planned and documented early, when requirements are being defined. Number of tests will increase as more end-to-end functionality becomes available. |
| Tools to use | Tools to use include:
Sample forms to exercise exposed functionality before GUI is available.
GUI record and playback tools. |
| Automation role | Automation is encouraged to enable multiple, quick Build Verification Tests on new software builds. |
Table 2: Functional Tests
| Succeeded/Failed | Test |
|---|
| | Application has visual display of offline and online status. |
| | Application allows user to log in through the Internet. |
| | Application allows user to log in through the corporation's network. |
| | Application prompts user when user logs in as to whether they want to be in offline or online state (applies only when the user is online). |
| | Application prompts user to synchronize work queue data when the user logs off (this only occurs if the user made changes to the work queue when offline). |
| | Test that application displays work items when it is online. |
| | Test that application displays work items when it is offline. |
| | Test that application changes work item that requires application to be online. |
| | Test that application prompts user for synchronization of work items for updated queue data. |
| | Test that user can manually force the system to go offline. |
| | Test that user can manually switch the system back to online. |
| | Test that application has automatic connection state detection strategy. |
| | Test that application has download interface for downloading work items. |
| | Test that user can download work items, go offline, update work items when offline, go online and sync back when in online. |
| | Test that application marks the data as dirty when edited and checks for expiration of stale data and dirty data. Only stale data should be overwritten from the server. |
| | Test to see if MSDE cache storage provider is used for caching data. |
| | Test to see if isolated cache storage provider is used for caching data. |
| | Test to see if Message Queuing queue storage provider is used for queuing data. |
| | Test to see if MSDE queue storage provider is used for queuing data. |
| | Test that multiple updates to client data when offline are synchronized when online. |
| | Test that multiple providers can be configured for ConnectionDetectionStrategy. |
| | Test that here is support for using different cache stores for ReferenceDataCache. |
| | Test that multiple providers can be configured for QueueProvider. |
| | Test the polling interval configuration for ConnectionDetectionStrategy. |
| | Test the polling interval configuration for Executor operation. |
| | Test the maximum queue messages configuration for QueueProvider. |
| | Test the expiration configuration for cached data. |
| | Test the connection string configuration for connecting to MSDE. |
| | Test that encryption using a custom provider is implemented through the ICryptographicProvider interface. |
| | Test the download frequency configuration property. |
| | Test error and boundary conditions. |
| | Test that it is possible to store and retrieve data across multiple running instances of the Offline Application Block. |
White Box Testing
White box testing focuses on small, specific areas of the code.
Table 3: White Box Testing Definition
| Area | Definition |
|---|
| Scope | Tests cover small units of code and are performed using code walkthroughs, code review, and unit-level tests that use a test framework. |
| When to start | Tests are conducted early in the development cycle. Tests should be performed as code is being developed and as soon as small, independent units of code are ready to be exercised. |
| Tools to use | Tools to use include:
Code-coverage tool
Memory profiling tool
Code review to see that code adheres to development guidelines
Unit-testing framework tool (should be able to automate) |
| Automation role | Unit test automation is crucial to performing repeatable, thorough tests on an application that is still under development. |
Table 4: White Box Tests
| Succeeded/Failed | Test |
|---|
| | Walk through the code and review it for such qualities as logic, adherence to development standards and best practices, and readable comments. |
| | Run the code against a code-coverage tool to make sure that all the available code is being tested. |
| | Perform memory profiling of the code to make sure that objects are correctly released and garbage is collected. |
| | Use unit-testing framework tool to create unit tests for testing public methods. Extend these tests and run them periodically as the code is developed. Unit tests act as quick checks to ensure that the code isn't broken as it changes. To make unit tests more logical, map them to the test cases described in the "Functional Testing" section. |
Security Testing
Security testing tests that the application and its data are protected and that privacy is protected and data properly encrypted.
Table 5: Security Testing Definition
| Area | Definition |
|---|
| Scope | Tests cover such areas as verifying that privacy is protected, that data is encrypted, that data is tamper-proof, and that the application can withstand various types of malicious attacks. Security testing should focus both on independent system units and on the overall application. |
| When to start | Tests should be planned when the requirements are being defined. Tests should be executed as security measures and methods are implemented. |
| Tools to use | Tools to use include:
DOS attacks simulation
Network sniffing tools
Also, refer to checklist chapter of |
| Automation | Not applicable |
Table 6: Security Tests
| Succeeded/Failed | Test |
|---|
| | Test that application securely downloads reference data to local computer. |
| | Test that application securely uploads the updated reference data to the server. |
| | Test that application restricts access to downloaded data to authorized users. |
| | Test that it's possible to secure access to services through company intranet. |
| | Test that it's possible to secure access to services through the Internet. |
| | Test that there is secure storage of downloaded messages. |
| | Track to see if downloaded reference data is tampered with. |
| | Test to see if there is restricted access to queued data through proper logon credentials. |
| | Test to see that queued data is properly encrypted and signed. |
| | Test to see that there is restricted access to cached data through proper logon credentials. |
| | Test to see that cached data is properly encrypted and signed. |
| | Test to see if it's possible to use service for downloading through 3rd party certification (such as Verisign). |
| | Test to see that assemblies are encrypted, digitally signed at deployment server, and validated on download at the client. |
| | Test to see that proxy generated for the service has valid end-points to prevent spoofing. |
| | Test to see that cached and queued data for multiple users sharing a client is secure. |
| | Verify that DemandReflectPermission is implemented properly in the code that uses reflection. |
Performance Testing
Performance testing tests the application's processing and response times under various conditions.
Table 7: Performance Testing Definition
| Area | Definition |
|---|
| Scope | Tests cover application processing and response performance under various data loads, memory stress conditions, network availability, and different connection speeds. |
| When to start | Tests should be planned when the requirements are being defined. Tests can be conducted against completed units by using in-house test stubs as well as performance test tools. Number of tests will increase as more end-to-end functionality becomes available. |
| Tools to use | Tools to use include:
|
| Automation | Automation is encouraged to enable multiple, quick performance tests on newer software builds. This will ensure that the application meets acceptable performance criteria and is also useful for comparison studies. |
Table 8: Performance Tests
| Succeeded/Failed | Test |
|---|
| | Test to see if downloading occurs over prolonged time period. |
| | Test to see if uploading occurs over prolonged time period. |
| | Test performance by downloading several hundred items in a row over time. |
| | Test performance by uploading several hundred items in a row over time. |
| | Test performance by downloading and uploading varying data sizes over time. |
| | Test performance by downloading and uploading data using different bandwidths and with delay times. |
| | Test performance when downloading and uploading data with limited memory and disk space. |
| | Analyze the following types of data:
System Pool resources
Contention
Process working sets
System queues
Process CPU context
Memory and IO
Network
System resources
Process availability
Exceptions
Process resources
Transactions times. |
Integration Testing
Integration testing ensures that the application works with other systems and components.
Table 9: Integration Testing Definition
| Area | Definition |
|---|
| Scope | Tests cover integration of application with external components and pre-existing systems. An example is to test the application's interaction with Web services. Tests verify the entire workflow, along with all the interactions between the various components that make up the application. |
| When to start | Tests should be planned when the requirements are being defined. Number of tests will increase as different parts of the system are integrated. |
| Tools to use | Not applicable. |
| Automation | Not applicable. |
Table 10: Integration Tests
| Succeeded/Failed | Test |
|---|
| | Test the integration between the client application code and Web services. |
| | Test the downloading of work items and subsequent uploading of modified work items. |
| | Test the queuing of work items when offline and the execution of the queued messages when back online. |
| | Verify the working of different queue providers in combination with the different cache providers. |
| | Apply multiple updates to the work items when offline and check the synchronization when back online. |
| | Test the approach to filtering reference data on the server side to enable downloading applicable parts of the database. |
| | Test that the application displays to the user a list of work items that collide with changes made by other users, based on server notifications. |
| | Test that the application synchronizes logs with server logs as per defined periods of time. |
| | Test collision scenarios where the same data is modified in the client as well as simultaneously being updated in the server. |
| | Test that clients are refreshed by adding new data (work items) on the server. |
| | Test that new data (work items) added on the client is synchronized with the server. |
| | Modify the schema of a work item in the server and update the client |
| | Test the scenario where two or more separate online clients simultaneously update data on the server. |
| | Test the scenario where two or more offline clients modify data and update the server. |
| | Ensure that test cases cover verification of the application's workflow on supported platforms. |
Content Testing
Content testing verifies that the documentation is correct.
Table 11: Content Testing Definition
| Area | Definition |
|---|
| Scope | Tests verify the content of instructions given to users during installation and operation. Tests cover supplemental user guides and help files. Tests verify usability, correctness, and inclusiveness of the content. |
| Tools to use | Content testing is largely done manually. Usability and test engineers who are familiar with the application are able to point out contradictions or missing information, as well as discrepancies between the application and the documentation. |
| Tools to use | Not applicable. |
| Automation | Not applicable. |
Table 12: Content Tests
| Succeeded/Failed | Test |
|---|
| | Test that content is technically correct. |
| | Test that the order of steps in terms of operations, flow, and chronology is correct. |
| | Test that the content is relevant. |
| | Test that the content is unambiguous and easy to follow. |
| | Test that the content is consistent and uniform. |
| | Test that the content provides complete information on each topic. |
| | Test that topics that need clarification are supported with appropriate examples for better understanding. |
| | Test that a topic has references to relevant and related topics. |
| | Test that content takes into account the technical level and familiar communication language of the expected user. |
Installation Testing
Installation testing verifies that the application installs correctly on the client machine.
Table 13: Installation Testing Definition
| Area | Definition |
|---|
| Scope | Tests verify functionality of application's final delivery and installation on the client machine as well as verifying delivery of all applicable files. |
| When to start | Tests are usually conducted at the end of the development and test cycles. |
| Tools to use | Tools to use include GUI record and playback tools (should be able to automate). |
| Automation role | Use build verification test (BVT) automation scripts created during functional testing to quickly verify main functionality of application after installation. This provides a quick way to verify that installations are correct. |
| Automation tools | GUI record and playback tools to conduct BVT tests against the installed application. The tool provides a quick way to verify installs. |
Table 14: Installation Tests
| Succeeded/Failed | Test |
|---|
| | Test that application installs, and that scripts run without errors and that all major functionalities pass tests. |
| | Test that all files install on client machine with correct versions (including both code and content). |
| | Test that it is possible to uninstall the application and test verification of the clean-up. |
| | Verify the naming standards of the files in the installation. |
| | Verify that the installer exits gracefully for error scenarios such as lack of disk space. |
| | Verify the registry entries during installation and the registry clean-up during uninstall. |
| | Perform a clean machine installation. This is a machine with a newly installed operating system and the minimum number of required pre-existing components. |
| | Test the installation on machines with different software configurations. |
| | Test that the installer creates the right Start menu entries. |
| | Test that the installer places files in the right folders. |
| | Test that assemblies are encrypted, digitally signed at deployment server, and validated on download at the client. |
Start | Previous