Testing Device Drivers on Windows Platforms
Device drivers are software programs that sit between the hardware and application software layers and are one of the most important subsystems in the computer hardware and software interaction process. The hardware abstraction layer of the operating system (OS) interacts directly with the device driver subsystem to send and receive commands between the application software layer and the actual device hardware.
To develop device drivers, you must understand the commands and timing that the hardware device uses and also understand the driver framework that the OS supports. In most cases software runs on the hardware to perform a specific task, and device drivers act as a pipeline to send or receive control and data messages to the hardware by using well-established protocols, such as PCI, PCI-Express, USB, and so on. Sometimes direct hardware manipulation, such as memory mapped I/O, is performed by the drivers directly, especially in embedded devices that run other versions of the Windows OS, such as Windows Embedded CE.
To test drivers, you must understand application software testing. However drivers are in a special class of software, because they can more freely access lower-level system resources than most application software can. You must be careful when you create a test plan and develop driver tests, because if a test fails, it may cause the system to stop responding and you might have to re-image the OS to recover.
Strategies for testing drivers can be generally categorized into the following areas:
Use automation wherever possible to catch the most number of bugs.
Test early to catch development bugs during the development phase.
Use a combination of configurations and device settings.
Use equivalence testing, such as grouping drivers into classes and then testing on that class. For example, one .sys file may represent the driver for 100 types of printers, where not all the printers in that class can be tested. In this situation, determining the class of printers affected by a driver change would be useful and might save time.
Run stress tests together with regular tests.
Incorporate aspects that are specific to the driver subsystem, not the application software subsystem, into functional tests.
Use established test resources such as Driver Test Manager (DTM) and the test framework from Windows Hardware Developer Central (WHDC).
Run Windows Logo tests and submit the result to the Windows Hardware Quality Labs (WHQL).
Perform scenario and integration tests based on customer data that target the same class of drivers.
The following methods are typically used to test device drivers.
A driver is a software program, and, like any software program, the source code has to be tested by several different analyzer and verifier tools to find code faults during development.
Static analysis can be conducted by tools such as PREfast for drivers. This tool is a static source code analyzer that detects certain classes of errors that are not detected by a compiler, such as checking for the correct interrupt request level (IRQL). This tool is available as a stand-alone tool in the Windows Driver Kit (WDK). A detailed guide for PREfast can be found at this Microsoft Web site.
Driver developers and testers also use Static Driver Verifier (SDV). This is a compile-time tool for Windows device drivers based on the Windows Driver Model (WDM). SDV places a driver in an unstable state and systematically tests all the code paths by looking for violations of WDM rules. This tool tests functions that use I/O request packets (IRP), interrupt request levels (IRQL), Plug and Play (PnP), power management, Windows Management Instrumentation (WMI), and so on. This tool is available in the WDK. For information about SDV, see this Microsoft Web site.
For kernel mode drivers, a powerful tool that developers and testers frequently use is Driver Verifier. It verifies that drivers are not making unauthorized function calls, which cause memory corruption, mishandling of I/O request packets, use of invalid Direct Memory Access buffers, and so on. Driver Verifier can be used as a graphical user interface (GUI) tool, a command-line tool, or as an extension to Debugging Tools for Windows. Driver Verifier is built into the Windows Vista operating system. More information about Driver Verifier can be found at this Microsoft Web site.
Debugging Tools for Windows is a set of extensible tools for debugging device drivers for the Microsoft Windows family of operating systems. Debugging Tools for Windows supports debugging of:
Applications, services, drivers, and kernels
Native 32-bit x86, Intel Itanium, and native x64 platforms
Windows NT 4.0, Windows 2000, Windows XP, Windows Server 2003, Windows Vista, and Windows Server 2008
User-mode and kernel-mode drivers
Live targets (local and remote) and dump files
This set of tools includes Windbg, a powerful GUI-based debugger, and the console-based debuggers: NTSD, CDB, and KD.
|Windbg is a "must use" tool for most types of driver testing outlined in this article. You run it from a host device that is connected to the target test device by a null-modem cable.|
The run-time analysis and debugging tools are useful when the driver is executing an operation. For more information about Debugging Tools for Windows, see this Microsoft Web site.
Functional testing applies to any software. For drivers, functional testing includes a suite of tests to check whether the behavior of the driver follows the design document for the driver. You run functional tests under regular conditions on an isolated test device that contains the driver and is minimally configured or customized. You should begin functional testing during the development phase as soon as the driver is usable and somewhat stable. You can use these tests repeatedly during subsequent phases of driver development and testing.
A driver might behave differently in different configurations. Therefore, you should conduct tests in several kinds of test environments and make configuration testing a part of the regular test pass. Configuration testing includes system configuration changes, system environment changes, and device settings changes:
Test the driver on several different platforms such as x86, x64, and Itanium, with Hyberfile, with a multiprocessor device and multicore devices, with different chipsets, and checked builds.
Test the driver with different memory configurations, from low memory to high memory. This testing might yield some unexpected errors that you might not catch otherwise. Another good test is raising the CPU utilization of another program while testing the driver. This test frequently yields timing bugs.
Test the driver with different device settings, ranging from default options to options that are rarely used. The least-used options might find unexpected errors, because this area generally receives the least attention during development. If the device uses hardware resources such as IRQ, I/O Port, and so on, the test matrix should include testing changes in these resources. Hardware compatibility testing is also important. This test consists of testing the driver on several devices that use different connection types, such as using PCI and PCI-Extender (if the hardware is PCI-based).
Scenario and Integration Testing
Scenario testing is a great way to test any software, whether it is application software, a database, or a driver. From historical customer data, you can generally infer how the driver will be used by the customer, and you should test those scenarios. Testing the functional behavior and using automation is a great way to find bugs. However, most unexpected errors are found by scenario and integration testing.
Integration testing also simulates the customer environment in some cases. For example, most customers log on as a user (instead of with administrator permissions) and install some software such as Microsoft Office, Windows Media Player or QuickTime, antivirus software, and so on. Testing the driver in an environment that simulates this scenario can find bugs not found during typical functional verification.
The objective of stress testing is to place the driver in an environment with unexpected or high load conditions. Stress testing drivers is very important. You perform stress testing by repeatedly using the same functions to find system failures and memory leaks. Stress testing is useful when run in combination with high CPU utilization by some other software.
Occasionally, developers and testers confuse long-haul testing with stress testing. The objective of long-haul testing is to simulate how a driver is used on a day-to-day basis in the end user’s environment. Long-haul testing runs applications over a long period of time while applying a typical driver load. This testing yields difficult-to-find bugs that are related to timing and which might not be discovered by typical testing.
Because drivers have higher privileges for accessing system resources, such as memory or PnP I/O, you should test drivers in ways that you typically do not test most application software. These additional tests include the following:
I/O cancel tests
Power management tests
You can find device driver test tools at this Microsoft Web site. The following list includes several tests that developers and testers typically use during the development and test cycle.
Windows Logo Program
These tests are intended to improve an end user's confidence in the devices he or she purchases by ensuring the devices and drivers meet certain standards. To obtain the Windows Logo for a driver, the driver must pass predefined tests. These tests also check the driver interface. For more information about the Windows Logo Program, see this Microsoft Web site.
Driver Test Manager
For a driver to qualify for the "Designed for Microsoft Windows" logo, it must pass testing by Driver Test Manager (DTM). More information about DTM is available at this Microsoft Web site.
Windows Device Test Framework
You can use the Microsoft Windows Device Testing Framework (WDTF) to create, manage, reuse, and extend device-centric, scenario-based automated tests. The following illustration shows a typical WDTF model for creating scenarios.
Detailed information about WTDF is available at this Microsoft Web site.
Device Simulation Framework
You can use Device Simulation Framework (DSF) to create a high-fidelity simulation of hardware in software. The simulation then appears like a real device to Windows. This tool is one of the newest frameworks for device simulation. More information is available at this Microsoft Web site.
Windows Error Reporting
The Windows Error Reporting tool provides a database of Windows errors and problems and lists solutions for many different issues. Details are available at this Microsoft Web site.
Additional Microsoft Test Resources
When you test drivers on Windows Embedded Standard 2009, you must perform one more layer of testing in addition to the general driver testing mentioned earlier.
In Windows Embedded Standard 2009, the dependency of the binary is the one of the most important considerations. Most of the drivers in Windows Embedded Standard 2009 come from the Windows team and other feature teams at Microsoft and have been rigorously tested. However, the drivers for Embedded Enabling Features are not created by other teams.
In Windows Embedded Standard 2009, Embedded Enabling Features are treated as components, which are the basic units in Windows Embedded Standard, not a file, as you might expect. Components are created from driver files, and developers must add the required dependency information to the component by using dependency tools such as Depends, which is available in Visual Studio 2008.
The componentization tester verifies whether all the dependency information is present in the component that owns the driver file. This testing includes metadata verification and a dependency check to verify that the Windows Embedded Standard 2009 run-time image pulls in the components that are required for the driver to function correctly. If the dependency information is incorrect, the driver might behave in an unexpected way.
Testing Embedded Enabling Features
File-Based Write Filter (FBWF), Enhanced Write Filter (EWF), USB Boot, and Registry Filter (RegFlt) are Embedded Enabling Features in Windows Embedded Standard 2009. These are all drivers. To test these Embedded Enabling Feature drivers, you must perform driver compliance tests, functionality verification tests, and end-to-end scenario tests.
Driver Compliance Tests
Each Embedded Enabling Feature must comply with the rules that apply to that particular type of driver. For example, FBWF is a file system mini-filter driver. Because it is a mini-filter driver, it must comply with whatever behavior Windows expects from a file system mini-filter driver.
These tests make sure that you do not violate any basic driver subsystem expectations. By not violating expectations, you help guarantee stability and application compatibility. Some tools that are used for testing driver compliance include the following:
These are source analyzer tools that can expose hidden issues.
These tools should be enabled whenever a driver runs. MSDN contains extensive documentation on these tools and their individual settings.
These tests are part of the Windows Logo Kit.
Basic IOCTL fuzzers
An example of this type of tool is a device path exerciser.
Fault injection tests
Functionality Verification Tests
Functional testing of Embedded Enabling Features targets several different test areas such as API testing, core functionality testing, boundary testing, stress testing, and long-haul testing that are specific to the way the target Embedded Enabling Feature is used.
End-to-End Scenarios Test
End-to-end scenario testing replicates common user or device driver tasks. These tests try to answer questions such as "Does the terminal client work on a thin client computer protected by FBWF?" The tests conducted in this area are generally very broad in scope.