Crashdump Support Test

Article
09/14/2012

Note This content applies to the Windows Logo Kit (WLK). For the latest information using the new Windows Hardware Certification Kit (HCK), see Windows HCK User's Guide on the Windows Hardware Dev Center.

Type: Automated Test

Overview

Crashdump is the mechanism in which the OS calls the storage adapter driver to write the contents of the memory to a dump file after a system stop error. However, due to the nature of a system stop error, the OS cannot make any assumptions about the stability of the system, and thus very few system services are available to the storage drivers.

The Crashdump Support Test verifies that the storage adapter driver can still operate under these constrained environments and perform the I/O to write to the dump successfully.

Details

The Windows OS allows three different types of memory dump files to be generated:

Full
Kernel
Mini

The test will test all three dump file types. For further information about these dump file types, please refer to the Microsoft Kernel Debugger documentation.

The test will go through the following sequence of operations:

Install Debugging Tools for windows to the %SystemDrive%\Debuggers directory and initialize the system.
Install test driver to simulate crash.
Test will simulate system stop errors (bluescreen) with bugcheck code 0x000000E2.
System reboots, the test will re-launch automatically.
Test will examine the previous crash by analyzing the memory dump file through the kernel debugger, and verify that the dump has been written correctly.
Repeat steps 3 to 5 for each dump file type.

Run Time: 30 minutes

Log File:

System Restart Required: Yes

Test Category:

Supported operating systems for Logo or Signature testing:

Windows 7
Windows Server 2008 R2
Windows Vista
Windows Server 2003
Windows XP

Program:

Requirements

Software Requirements

The test tool requires the following software:

Supported operating system (see list above)

Hardware Requirements

The test tool requires the following hardware:

Device to be tested
Hard drive with a minimum of 20 GB available on partition C:

Processor

Running Crashdump Support Test

Set the page file size on the test machine to at least the of size of physical memory + 1 MB.
Download and install Windows OS symbols using the detailed instructions below (symbols are needed to analyze the dump file; failure to install symbols is the most common test setup problem that would cause this test to fail, and you should not continue unless this issue is resolved). The version of the symbols must match the version of OS installed on the test machine, thus, it must be downloaded external to the kit.
Run test as DTM job.

Detailed Instruction on Installing Symbols:

Install the Windows Debugging Tools to c:\debuggers Debugging Tools for Windows - Overview
Open a command prompt (in Vista right-click and choose run as administrator).
Assuming the system directory is c:\windows\system32. Type in the following command: c:\debuggers\symchk /r c:\windows\system32 /s SRV*c:\symbols\*https://msdl.microsoft.com/download/symbols

This will download the symbols using the remote symbol store from Microsoft to the local directory c:\symbols, as the downstream store to cache the symbols. Please see this KB article for additional reference information: Use the Microsoft Symbol Server to obtain debug symbol files
Make sure that symbols can be found for OS kernel. Type in the following command:

c:\debuggers\symchk c:\windows\system32\ntoskrnl.exe /s c:\symbols\

Or, for PAE systems, type the following command:

c:\debuggers\symchk c:\windows\system32\ntkrnlpa.exe /s c:\symbols\

You should see an output similar to the following to confirm a successful symbols installation:

SYMCHK: FAILED files = 0

SYMCHK: PASSED + IGNORED files = 1
It is recommended that you uninstall Windows Debugging Tools before launching the test in DTM.

Command Syntax

Command option	Description
crashdumptest.exe -y <symbols_path> -dtm

Troubleshooting

The following is a list of common test failures categories, reference information, and methods to troubleshoot:

The test logs error message "Paging file size is too small for full dump purposes":

In the case of a crash, there is no certainty of what parts of the operating system will still be functional. The network or file system drivers may have caused the crash, for example, preventing access to file system structures to create a dump file, or network to store the file remotely. The OS handles this by using a file that it already knows exists (the pagefile) and writes directly to that file's logical block extents on disk. The dump process writes the contents of physical memory into the pagefile on the system disk (usually c:\pagefile.sys). The pagefile must be large enough to contain the dump. The largest dump is a complete dump which requires the size of physical memory (ex: 4096) plus one extra megabyte to contain the header information. The test requires that the user configure the page file to an appropriate size before execution (see Windows does not save memory dump file after a crash). If the page file size is insufficient, the test will log the following error during the initialization phase:
- (i) Verifying paging file size.
- (x) Paging file size is too small for full dump purposes.
- (i) Paging file size: 330989568
- (i) Physical memory size: 1073094656
- (i) Please configure minimum paging file size to physical RAM size + 1MB.
I see the system stop error (blue screen), but the bugcheck code is not E2:

After some basic settings validation the test will install a driver used to crash the system and reboot the system. After the reboot the test changes the crash control settings (for full memory dump), deletes any old dump files, and crashes the system. At the point of the crash the system will display a bugcheck screen (blue screen) with details of the nature of the crash. The type of bugcheck should be MANUALLY_INITIATED_CRASH (e2). If anything else appears here it means a second bugcheck occurred during the process of writing the dump file. This should be investigated by connecting a kernel debugger to the test client and debugging the storage adapter driver.
The test logs error message "Dump file is being used by another process. HRESULT: 0x80070020":

After the dump file has been written the test machine should automatically reboot. Upon boot after a crash the operating system will detect the presence of dump information in the page file and begin the process of writing a dump. This process occurs asynchronously while the machine is booting and even after the user has logged in (see What to consider when you configure a new location for memory dump files in Windows Server 2003 for details). During this process you can view it's progress by checking the size of the dump file (C:\windows\memory.dmp) or viewing the process in task manager (werfault.exe). The test will often be running at the same time as this process, trying to access the same dump file. If this occurs the following messages will appear in the log:
- (i) Connecting to DumpFile: C:\Windows\MEMORY.DMP
- (i) Dump file is being used by another process. HRESULT: 0x80070020
- (i) Usually memory.dmp is still being written due to large RAM, retry after 5 minutes.
The test should then retry accessing the file. If the error message code is different, or if it changes (ex: 0x80070002 : ERROR_FILE_NOT_FOUND) then it means the file could not be written to disk. The first place to check for valuable debug information is the system event log. To view the event log click Start > Run > Compmgmt.msc. In the computer management window, select Computer Management\System Tools\Event Viewer\System. Browse through the list of events for an event with the source BugCheck. The most common cause for a missing dump file is insufficient free space on the disk. As noted in the KB article referenced above, the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\MachineCrash contains information about the last crash (before a reboot), including the partial dump file if one was created. This can be useful when trying to debug other missing dump issues.
The test logs error message "Failed to load the correct symbols":

Upon reboot the test examines the dump file for correctness. The test does this, just as a developer would, using the kernel debugger kd ("Debugging Tools for Windows" package). In order to analyze a dump file the debugger needs access to symbols. I won't go into details on why these are needed (more information is available here), but think of these as a dictionary for the dump. They allow the debugger to analyze the contents of memory (or a crashdump file) into individual modules (executables, libraries, drivers, etc), functions within those modules, and data structures.

For the test to work properly, it needs to provide the debugger with symbols. When it does not have proper symbols it will log warnings during the log failures during the first analysis phase of testing. The current mechanism for doing so is to have the user download public symbol packages (here) and install them on the test machine before running the test. When symbols are not installed or the symbols do not match the operating system under test, the following message may appear in the test log:
- (x) Failed to load the correct symbols.
- (i) Please refer to WDK documentation on how to install OS symbols.
- (i) Your symbols might also be out of date, please update symbols via the symbol server: Use the Microsoft Symbol Server to obtain debug symbol files
This won't actually cause the test to fail because in some cases the dump can still be analyzed with partially matching symbols. If the test continues and more test cases fail with messages like "Error retrieving addresses of ..." or "Unable to get..." it means that the debugger cannot analyze the dump due to the missing symbols. We understand the difficulties associated with downloading the public symbol packages and ensuring they are up to date with all of the latest hot-fixes on the system. One way we have found to work around a symbol is to supplement the locally installed symbol packaged with symbols cached from an internet symbol server. The test could use the symbol server directly, but it is not advised to have the test machines connected to the internet when under test. The steps for caching the symbols locally are as follows:
1. Ensure that you have already created a crashdump. The easiest way to do this is to run the test once and let it fail.
2. Ensure that you have the debugging tools installed. Again the easiest way to do this is to run the test once. It will install the tools to C:\Debuggers.
3. Open a command prompt (in Vista right-click and choose run as administrator).
4. Type in the following command, without the quotes:
  
  "**c:\Debuggers\kd -z c:\Windows\MEMORY.DMP -y SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols**"
5. This will load the dump in the kernel debugger using the remote symbol store at Microsoft and the local directory C:\Symbols as the downstream store to cache the symbols.
6. Make sure that symbols can be found for OS files like NTOSKRNL and NTDLL. These are necessary to analyze the dump. It is ok if errors appear loading symbols for other modules like 3rd party drivers.
7. You should now have a prompt "0: kd>". At this prompt type the command ".reload /f" without the quotes. This command forces the debugger to load and cache all symbols for modules loaded in the dump.
8. You can now exit the debugger with CONTROL+B then enter.

Code Tour

File Manifest

File	Location
crashdumptest.exe	...\tests\%ARCHITECTURE\nttest\driverstest\storage\wdk