Testing Device Drivers on Windows Platforms

4/24/2012

Microsoft Corporation

May 2009

Summary

This technical article contains information about test strategies, methods, and resources for device drivers in Windows Embedded Standard 2009.

Introduction

Test Strategies

Test Methodologies

Test Resources

Testing Drivers in Windows Embedded Standard 2009

Conclusion

Introduction

Device drivers are software programs that sit between the hardware and application software layers and are one of the most important subsystems in the computer hardware and software interaction process. The hardware abstraction layer of the operating system (OS) interacts directly with the device driver subsystem to send and receive commands between the application software layer and the actual device hardware.

To develop device drivers, you must understand the commands and timing that the hardware device uses and also understand the driver framework that the OS supports. In most cases software runs on the hardware to perform a specific task, and device drivers act as a pipeline to send or receive control and data messages to the hardware by using well-established protocols, such as PCI, PCI-Express, USB, and so on. Sometimes direct hardware manipulation, such as memory mapped I/O, is performed by the drivers directly, especially in embedded devices that run other versions of the Windows OS, such as Windows Embedded CE.

To test drivers, you must understand application software testing. However drivers are in a special class of software, because they can more freely access lower-level system resources than most application software can. You must be careful when you create a test plan and develop driver tests, because if a test fails, it may cause the system to stop responding and you might have to re-image the OS to recover.

Test Strategies

Strategies for testing drivers can be generally categorized into the following areas:

• Use automation wherever possible to catch the most number of bugs.
• Test early to catch development bugs during the development phase.
• Use a combination of configurations and device settings.
• Use equivalence testing, such as grouping drivers into classes and then testing on that class. For example, one .sys file may represent the driver for 100 types of printers, where not all the printers in that class can be tested. In this situation, determining the class of printers affected by a driver change would be useful and might save time.
• Run stress tests together with regular tests.
• Incorporate aspects that are specific to the driver subsystem, not the application software subsystem, into functional tests.
• Use established test resources such as Driver Test Manager (DTM) and the test framework from Windows Hardware Developer Central (WHDC).
• Run Windows Logo tests and submit the result to the Windows Hardware Quality Labs (WHQL).
• Perform scenario and integration tests based on customer data that target the same class of drivers.

Test Methodologies

The following methods are typically used to test device drivers.

Static Analysis

A driver is a software program, and, like any software program, the source code has to be tested by several different analyzer and verifier tools to find code faults during development.

Static analysis can be conducted by tools such as PREfast for drivers. This tool is a static source code analyzer that detects certain classes of errors that are not detected by a compiler, such as checking for the correct interrupt request level (IRQL). This tool is available as a stand-alone tool in the Windows Driver Kit (WDK). A detailed guide for PREfast can be found at this Microsoft Web site.

Driver developers and testers also use Static Driver Verifier (SDV). This is a compile-time tool for Windows device drivers based on the Windows Driver Model (WDM). SDV places a driver in an unstable state and systematically tests all the code paths by looking for violations of WDM rules. This tool tests functions that use I/O request packets (IRP), interrupt request levels (IRQL), Plug and Play (PnP), power management, Windows Management Instrumentation (WMI), and so on. This tool is available in the WDK. For information about SDV, see this Microsoft Web site.

Run-Time Analysis

For kernel mode drivers, a powerful tool that developers and testers frequently use is Driver Verifier. It verifies that drivers are not making unauthorized function calls, which cause memory corruption, mishandling of I/O request packets, use of invalid Direct Memory Access buffers, and so on. Driver Verifier can be used as a graphical user interface (GUI) tool, a command-line tool, or as an extension to Debugging Tools for Windows. Driver Verifier is built into the Windows Vista operating system. More information about Driver Verifier can be found at this Microsoft Web site.

Debugging Tools for Windows is a set of extensible tools for debugging device drivers for the Microsoft Windows family of operating systems. Debugging Tools for Windows supports debugging of:

• Applications, services, drivers, and kernels
• Native 32-bit x86, Intel Itanium, and native x64 platforms
• Windows NT 4.0, Windows 2000, Windows XP, Windows Server 2003, Windows Vista, and Windows Server 2008
• User-mode and kernel-mode drivers
• Live targets (local and remote) and dump files

This set of tools includes Windbg, a powerful GUI-based debugger, and the console-based debuggers: NTSD, CDB, and KD.

The run-time analysis and debugging tools are useful when the driver is executing an operation. For more information about Debugging Tools for Windows, see this Microsoft Web site.

Functional Testing

Functional testing applies to any software. For drivers, functional testing includes a suite of tests to check whether the behavior of the driver follows the design document for the driver. You run functional tests under regular conditions on an isolated test device that contains the driver and is minimally configured or customized. You should begin functional testing during the development phase as soon as the driver is usable and somewhat stable. You can use these tests repeatedly during subsequent phases of driver development and testing.

Configuration Testing

A driver might behave differently in different configurations. Therefore, you should conduct tests in several kinds of test environments and make configuration testing a part of the regular test pass. Configuration testing includes system configuration changes, system environment changes, and device settings changes:

• System Configuration
  Test the driver on several different platforms such as x86, x64, and Itanium, with Hyberfile, with a multiprocessor device and multicore devices, with different chipsets, and checked builds.
• System Environments
  Test the driver with different memory configurations, from low memory to high memory. This testing might yield some unexpected errors that you might not catch otherwise. Another good test is raising the CPU utilization of another program while testing the driver. This test frequently yields timing bugs.
• Device Settings
  Test the driver with different device settings, ranging from default options to options that are rarely used. The least-used options might find unexpected errors, because this area generally receives the least attention during development. If the device uses hardware resources such as IRQ, I/O Port, and so on, the test matrix should include testing changes in these resources. Hardware compatibility testing is also important. This test consists of testing the driver on several devices that use different connection types, such as using PCI and PCI-Extender (if the hardware is PCI-based).

Scenario and Integration Testing

Scenario testing is a great way to test any software, whether it is application software, a database, or a driver. From historical customer data, you can generally infer how the driver will be used by the customer, and you should test those scenarios. Testing the functional behavior and using automation is a great way to find bugs. However, most unexpected errors are found by scenario and integration testing.

Integration testing also simulates the customer environment in some cases. For example, most customers log on as a user (instead of with administrator permissions) and install some software such as Microsoft Office, Windows Media Player or QuickTime, antivirus software, and so on. Testing the driver in an environment that simulates this scenario can find bugs not found during typical functional verification.

Stress Testing

The objective of stress testing is to place the driver in an environment with unexpected or high load conditions. Stress testing drivers is very important. You perform stress testing by repeatedly using the same functions to find system failures and memory leaks. Stress testing is useful when run in combination with high CPU utilization by some other software.

Long-Haul Testing

Occasionally, developers and testers confuse long-haul testing with stress testing. The objective of long-haul testing is to simulate how a driver is used on a day-to-day basis in the end user’s environment. Long-haul testing runs applications over a long period of time while applying a typical driver load. This testing yields difficult-to-find bugs that are related to timing and which might not be discovered by typical testing.

Privilege Testing

Because drivers have higher privileges for accessing system resources, such as memory or PnP I/O, you should test drivers in ways that you typically do not test most application software. These additional tests include the following:

• Memory tests
• I/O cancel tests
• Setup tests
• Power management tests
• PnP tests
• Concurrency tests
• Penetration tests
• Security tests

Test Resources

You can find device driver test tools at this Microsoft Web site. The following list includes several tests that developers and testers typically use during the development and test cycle.

Windows Logo Program

These tests are intended to improve an end user's confidence in the devices he or she purchases by ensuring the devices and drivers meet certain standards. To obtain the Windows Logo for a driver, the driver must pass predefined tests. These tests also check the driver interface. For more information about the Windows Logo Program, see this Microsoft Web site.

Driver Test Manager

For a driver to qualify for the "Designed for Microsoft Windows" logo, it must pass testing by Driver Test Manager (DTM). More information about DTM is available at this Microsoft Web site.

Windows Device Test Framework

You can use the Microsoft Windows Device Testing Framework (WDTF) to create, manage, reuse, and extend device-centric, scenario-based automated tests. The following illustration shows a typical WDTF model for creating scenarios.

Detailed information about WTDF is available at this Microsoft Web site.

Device Simulation Framework

You can use Device Simulation Framework (DSF) to create a high-fidelity simulation of hardware in software. The simulation then appears like a real device to Windows. This tool is one of the newest frameworks for device simulation. More information is available at this Microsoft Web site.

Windows Error Reporting

The Windows Error Reporting tool provides a database of Windows errors and problems and lists solutions for many different issues. Details are available at this Microsoft Web site.

Additional Microsoft Test Resources

• Windows Hardware Developer Central (WHDC) is the main source for all driver-related information.
• For WDK and developer tools, see this Microsoft Web site.
• For testing tools for drivers, see this Microsoft Web site.
• For driver-related blogs, see this Microsoft Web site.

Testing Drivers in Windows Embedded Standard 2009

When you test drivers on Windows Embedded Standard 2009, you must perform one more layer of testing in addition to the general driver testing mentioned earlier.

Dependency Testing

In Windows Embedded Standard 2009, the dependency of the binary is the one of the most important considerations. Most of the drivers in Windows Embedded Standard 2009 come from the Windows team and other feature teams at Microsoft and have been rigorously tested. However, the drivers for Embedded Enabling Features are not created by other teams.

In Windows Embedded Standard 2009, Embedded Enabling Features are treated as components, which are the basic units in Windows Embedded Standard, not a file, as you might expect. Components are created from driver files, and developers must add the required dependency information to the component by using dependency tools such as Depends, which is available in Visual Studio 2008.

The componentization tester verifies whether all the dependency information is present in the component that owns the driver file. This testing includes metadata verification and a dependency check to verify that the Windows Embedded Standard 2009 run-time image pulls in the components that are required for the driver to function correctly. If the dependency information is incorrect, the driver might behave in an unexpected way.

Testing Embedded Enabling Features

File-Based Write Filter (FBWF), Enhanced Write Filter (EWF), USB Boot, and Registry Filter (RegFlt) are Embedded Enabling Features in Windows Embedded Standard 2009. These are all drivers. To test these Embedded Enabling Feature drivers, you must perform driver compliance tests, functionality verification tests, and end-to-end scenario tests.

• Driver Compliance Tests
  Each Embedded Enabling Feature must comply with the rules that apply to that particular type of driver. For example, FBWF is a file system mini-filter driver. Because it is a mini-filter driver, it must comply with whatever behavior Windows expects from a file system mini-filter driver.
  These tests make sure that you do not violate any basic driver subsystem expectations. By not violating expectations, you help guarantee stability and application compatibility. Some tools that are used for testing driver compliance include the following:
  • Prefix/PREfast
    These are source analyzer tools that can expose hidden issues.
  • Verifier/GFlags
    These tools should be enabled whenever a driver runs. MSDN contains extensive documentation on these tools and their individual settings.
  • Power management
    These tests are part of the Windows Logo Kit.
  • Basic IOCTL fuzzers
    An example of this type of tool is a device path exerciser.
  • Fault injection tests
• Functionality Verification Tests
  Functional testing of Embedded Enabling Features targets several different test areas such as API testing, core functionality testing, boundary testing, stress testing, and long-haul testing that are specific to the way the target Embedded Enabling Feature is used.
• End-to-End Scenarios Test
  End-to-end scenario testing replicates common user or device driver tasks. These tests try to answer questions such as "Does the terminal client work on a thin client computer protected by FBWF?" The tests conducted in this area are generally very broad in scope.

Conclusion

What you have learned

By using the information in this article, you should be able to better devise strategies and methods, and locate resources, for testing device drivers in Windows Embedded Standard 2009.