Stress Testing: Using Modular Stress Test to Test Platforms and Components

  • Article

 

Scott Baisch
Subhasis Mandal
Microsoft Corporation

December 2003

Applies to:
    Microsoft® Windows® CE .NET

Abstract

This document assumes that you are already familiar with the steps to build your own configuration with the Windows CE**Test Kit (CETK) component. You should also be familiar with the steps to run CETK and establish a connection to your target device.

Contents

Modular Stress
Types of Stress Testing
Modifying the Stress Environment
Running Modular Stress Test
Understanding Modular Stress Test Results
Running Modular Stress with Custom Module

Modular Stress

Modular Stress is a flexible system that performs several different types of stress testing. It is based on a set of test modules that each target individual features or components in a platform. The CETK includes a number of operating system (OS) stress modules that target major system components (for example, GWES, FileSys, Kernel, and so on). In addition, the user can create custom modules to target other platform components. The Modular Stress Test harness controls how these tests are run, that is, the sort of test environment that is desired, and collects data on system health throughout the course of a stress run.

The Stress Environment

The stress environment, that is the type and intensity of a stress, is governed by a number of parameters. The most important are:

  1. Module "Mix": the set of component test modules that are to be included in the stress run.
  2. Number of test modules that are to be run concurrently.
  3. The Module duration: the amount of time that each test module will run before it terminates and is replaced by another.

Stress Data Collection

The Modular Stress Test harness monitors the stress run and collects and reports data regarding system health:

  1. Individual module test case results
  2. System memory state
  3. Hang detection

The data is written to an Extensible Markup Language (XML) file on the device. For more information, see Understanding Modular Stress Results.

The Stress Process

The steps for running stress are outlined in the Running Modular Stress Test section and are described in detail in the CETK documentation. The following is a description of the Modular Stress process itself:

  1. The user sets the run parameters, for example:
    1. Test duration (default 900 minutes)
    2. Module duration (default 5 minutes)
    3. Number of concurrent modules (default 4)
    4. Reporting interval (default 10 minutes)
  2. The Modular Stress harness determines, which of the available modules, the targeted platform can support.
  3. The harness launches the specified number of concurrent modules, drawn at random from the list of supported modules.
  4. When a module reaches the end of its requested duration the harness replaces it with a new module, chosen at random.
  5. At a user defined interval (default 10 min):
    1. Modular Stress collects system memory stats and module results.
    2. Run data is written to a local XML file.
    3. Modular Stress checks to see if system has exceeded a user-defined low memory threshold.
    4. Modular Stress checks for hung modules.
    5. If either of the two preceding conditions are met, Modular Stress takes one of the following actions:
      1. Breaks into the debugger (default).
      2. Terminates the hung module and continue.

Types of Stress Testing

The Microsoft® Windows®CETest Kit provides a Modular Stress Test harness that allows the user to perform several different types of stress testing.

Component Integration Testing

Component integration testing tests multiple components operating concurrently in random combinations. This form of stress testing is typically very aggressive and produces situations that are difficult to produce in static test scenarios.

This form of stress testing targets:

  • Resource competition, for example:
    • Deadlocks
    • Starvation
    • Bottlenecks
  • Timing issues, race conditions, and so on.
  • Resource leaks

Mean Time to Failure (MTTF) Testing

There are a number of different definitions of Mean Time to Failure testing; however, for the purposes of Modular Stress, we consider this to be a stress test that targets typical user scenarios rather than lower level component integration. The goal is to estimate the durability and health of the system under standard usage patterns (usually accelerated) for the targeted device.

Individual Component Testing

Modular Stress can also be used to test the durability of a single component under constant use. This type of testing generally targets resource leaks and gradual functional degradation.

Modifying the Stress Environment

The CETK also assists you in modifying the stress environment to test against device-specific scenarios.

Component Integration Testing

This is the default environment. Component Integration testing typically involves all of the applicable test modules with the maximum number of concurrent modules. The goal is to produce an intense environment with the maximum number of combinations of components in competition for system resources.

It can be useful to vary the module duration between runs to produce different types of component tests:

  • Short module duration places a greater emphasis on load/unload, connect/disconnect, initialize clean-up operations.
  • Longer module durations more easily expose the accumulation of resources and gradual degradation of a component.

Mean Time to Failure (MTTF) Testing

Because this type of testing is driven by user scenarios it is device-specific. It is therefore up to the platform tester to define the scenarios and to create custom stress modules to simulate them. Extensive documentation regarding custom module development can be found in the CETK.

The stress environment should reflect expected usage:

  • Number of concurrent modules should correspond to user scenarios
  • Module duration should reflect typical usage (though it will likely be compressed by the omission of most idle time)

Individual Component Testing

This has been done with all of the OS modules provided with Modular Stress as part of a rigorous acceptance pass. It is recommended that this process be followed for custom modules before adding them to a larger mix. For example, it is much easier to isolate test problems outside of the chaos that integration stress can generate.

Each new module should be run in at least three different configurations before adding to a larger mix:

  1. Singly with a short duration for each instance (to test load/unload, create/destroy scenarios)
  2. Singly with a long duration (to check for leaks and long-term degradation)
  3. Concurrently with another instance of itself (to flush out component- or test-specific resource conflicts)

Running Modular Stress Test

There are two ways to run Modular Stress Test. It can be launched by using both the CETK interface and the Target Control command prompt. Running from the command prompt allows additional customizations and total control of the run as you can use all the available command-line options.

For option details, see Modular Stress Test Command-Line Options in CETK documentation.

Running Modular Stress Test from CETK

Follow these steps to run Modular Stress Test from the CETK interface.

  1. Connect the target device to CETK. For information about connecting a target device to the CETK, see Configuring the CETK to Communicate with a Target Device in CETK Help. After the connection is established, the Windows CE Test Kit window on the desktop computer should list your target device.

Figure 1.

  1. In the Windows CE Test Kit window, expand the node that corresponds to the target device.
  2. Expand the Other Tests node, and then select the Modular Stress Test check box.
  3. If you want, you can modify the run duration or other the behavior of the test by editing the command line for the test. For information about modifying the command line, see Editing the Command Line for a Test in CETK Help. For information about other command-line options for the Modular Stress Test, see Modular Stress Test Command-Line Options in CETK Help.
  4. Right-click Modular Stress Test, and then click Quick Start.

Figure 2.

After you start the Modular Stress Test, you should see a significant amount of activity on the display of the target device. If you have connected an instance of Platform Builder to the target device, the Output window in the Platform Builder integrated development environment (IDE) should display information about execution of the test modules.

Viewing Modular Stress Test Results from CETK

Unless a custom duration was specified by modifying the command line, CETK marks the test as complete after a successful run of 900 minutes. Choosing View Results from the context menu displays the result in Microsoft Internet Explorer.

Figure 3.

If CETK does not mark 'Modular Stress Test' as 'Complete' after the requested duration (default is 900 minutes), you must run Modular Stress Test with Platform Builder attached in a debugger-enabled image for additional debugging information. For more information, refer to the sections titled 'Running the Modular Stress Test from the Command Line' and 'Modular Stress Test Command-Line Options' in CETK Help.

Modular Stress Test from the Command Line

Running Modular Stress Test from the Command Line

The best choice of command line for the Modular Stress Test varies depending upon the setup of the target device. You may be able to more easily experiment with different command-line options if you run the Modular Stress Test from a command prompt rather than running the test from the CETK.

It is possible to run Modular Stress Test for a variety of test scenarios when launched from command line. This document discusses only the scenario where the target device is connected to a development workstation that is running Platform Builder. Other scenarios are described in the Running the Modular Stress Test from the Command Line section of the CETK Help. For detailed information about the command-line options for the Modular Stress Test, see Modular Stress Test Command-Line Options.

  1. Copy all files from <Platform Builder installation path>\Cepb\Wcetk\Ddtk\<CPU> to the release directory.
  2. Copy <Platform Builder installation path>\Cepb\Wcetk\Results.xslt to the release directory.
  3. At the command prompt, type oemstress -result -history -autooom

An oemstress.ini file must exist in the release directory on the development workstation. The harness creates a result.xml results file and a history.csv history file in the release directory.

Viewing Modular Stress Test Results from the Command Line

Using the preceding command line, the harness creates a file named result.xml to the release directory that contains the results for the stress run. The contents of this file gets updated every five minutes (this interval is customizable) and can be directly viewed in Internet Explorer in the file named results.xslt. Refer to step 2 in Running Modular Stress Test from the Command Line in this document.

Understanding Modular Stress Test Results

The results file for the Modular Stress Test shows the following information:

  • A list of missing files, if any files are missing (if launched from CETK)
  • The text OEMStress was completed successfully upon successful completion of the testing run
  • Information about how long the Modular Stress Test has been running
  • If a test module runs for much longer than specified duration, the text OEMStress has detected module hang displays. The Hang Status table shows detailed information about the hang
  • The total memory usage, if memory usage exceeds the specified threshold
  • A table named 'Launch Parameters' that displays the parameters used to run the Modular Stress Test
  • A table named 'System Details' that displays information about the target device such as CPU name, platform name, OS version, and so on
  • A table named 'System Health' that displays potential resource leaks. You can use a history file to view resource usage over time for more accurate leak detection. Please refer to notes at the bottom of this section for more information
  • A table named 'Hang Table' that displays a list of modules currently considered to be hung by the harness. This table does not appear if the hang detection option is disabled or if a hang is not detected
  • A table named 'Module Status' that displays pass/fail status of each test module

The following conditions must be satisfied to consider a stress session as passed:

  • The top of the result file should read, "OEMStress was completed successfully"

  • The top of the result file also displays how long OEMStress has been running. To consider a pass, Modular Stress Test should run for at least 900 minutes

  • The Module Status table, which lists pass/fail/warning information, the success rate above 99 percent should be displayed for all modules.

    Note   'Probable Leak' in the 'System Health' table by itself is not an accurate indication of a resource leak in the system. To get an accurate indication, follow these steps:

  1. Run Modular Stress Test from the command line with a duration of about 15 minutes. Allow all the stress modules terminate normally. This step establishes a realistic baseline for further resource usage tracking.
  2. Run Modular Stress Test with your command line with the duration you had in mind. After termination of this run, 'Probable Leak' in 'System Health' table will be much more accurate.

Running Modular Stress with Custom Module

Structure of oemstress.ini

Modular Stress stores module information in a simplified INI file with the following structure:

mod_name=stressmodule.dll
cmd_line=-f somefile.txt
dependency=stressutils.dll*stressmod.exe
os_comp=(COREDLL_CORECRT & COREDLL_COREMAIN & GWES_GWESMAIN & 
COREDLL_LMEM & ((GWES_MGRAST2 & GWES_MGRAST) | GWES_MGTT))
min_runtime=1

mod_name=stressmodule_custom.dll
cmd_line=-f somefile2.txt
dependency= 
os_comp= 
min_runtime=2

mod_name
(Required) Name of the stress module (complete file name) which can be either a .dll or .exe file.

cmd_line
(Required) Command line of for the stress module (excluding the module file name).

dependency
(Optional) List of files separated by '*' character the stress module is dependent on. When launching from CETK, the harness verifies the presence of the files at the device root. Leave this entry blank (as shown in the preceding example) for the custom modules.

os_comp
(Optional) OS component dependency information for the stress module. Leave this entry blank (as shown in the preceding example) for the custom stress modules.

min_runtime
(Required) Minimum amount of time (in minutes) the harness will request the module to run when launching.

Entries of the consecutive modules must be separated by single blank line.

Note   Commenting inside oemstress.ini is not supported.

Adding Custom Module to oemstress.ini

  1. Make a backup of <Platform Builder installation path>\Cepb\Wcetk\Ddtk\<CPU>\oemstress.ini.

  2. Append an entry for your module to the oemstress.ini located at the location specified previously:

    mod_name=stress_module.dll
cmd_line=cmd_line
dependency=
os_comp=
min_runtime=5
    • Replace stress_module.dll with the actual file name of the module.
    • Replace cmd_line with the appropriate command line for your module. Leave this entry blank if there is none.
    • Replace min_runtime with the minimum desired run time for the module. The harness will launch your module at least for this duration.
    • Other fields, dependency and os_comp must be left blank as shown.
    • Make sure that the consecutive modules are separated by single blank line.

  3. Save the modified oemstress.ini at <Platform Builder installation path>\Cepb\Wcetk\Ddtk\<CPU>.

  4. If you are planning to run Modular Stress from the command line, ignore this step and refer to Modular Stress Test from Command Line in this document. If you want to run Modular Stress with custom module from CETK, use Reedit and navigate to [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CETT\WinCE 4.20\Tests\Modular Stress Test] and add an empty key named after each of the test files (test modules, helper files).

Figure 4.

Refer to Modular Stress Test from CETK in this document to run Modular Stress from CETK.

Conclusion

The Modular Stress Test provides an environment where parts of a Windows CE .NET–based platform are exercised for an extended period of time. The Modular Stress Test verifies that a component is functionally stable after prolonged use, interacts appropriately with other components in arbitrary combinations and helps you to identify whether components deadlock, time out, or fail to function when competing for resources.

For more information about Test Tools for Windows CE .NET, visit the Windows CE .NET Test Tools page.