Bugslayer
Mini Dump Snapshots and the New SOS
John Robbins
Code download available at:Bugslayer0503.exe(126 KB)
Contents
Mini Dump Theory and Tools
ADPlus Usage
What's New in SOS
Wrap-Up
Tips
In debugging some large Microsoft® .NET Framework-based ap-plications over the last few months, I've been spending more time looking at mini dumps than at live processes. This is mainly because in those large applications problems surface when the apps are running in production and not on test systems. Because these mini dumps are so important, I've been surprised that many developers don't know how to create them.
In this Bugslayer installment, I will discuss the steps and tools necessary to snap mini dumps for .NET-based applications so you can debug them with the wonderful new Son of Strike (SOS) WinDBG extension DLL. With mini dumps of your .NET processes, you'll finally be able to stand a fighting chance at debugging those production-only problems.
I'll start out by discussing very briefly the type of mini dumps you'll need along with the appropriate tools. For most of the column, I'll show you the exact settings I use to get the proper mini dumps and some techniques for production mini dump snapping. Later I'll discuss some of the new features in the latest SOS Microsoft has released. I previously wrote about SOS (Bugslayer: SOS: It's Not Just an ABBA Song Anymore), but some of the new features can make tracking down tough .NET memory problems a breeze.
Mini Dump Theory and Tools
One thing that you may find confusing about mini dump snapshots is that their file size can be quite large. That's because SOS needs all the virtual memory pages marked as committed in order to do any of its magic. For a simple "Hello World" Windows® Forms application, the resulting mini dump is approximately 30MB. With ASP.NET applications, you're going to see mini dump sizes that range from 300MB to well over 1GB. While you can ask the end user to e-mail you the mini dumps, that will probably make their e-mail administrators and yours want to take mini dumps of both of you as well. At the very least, if you are going to transfer mini dump files around, compressing them with a tool like WinZip can reduce the file size by around 75 percent. Note that with the .NET Framework 2.0, you'll have the option of taking a smaller mini dump (on the order of a few hundred KB) which will allow you to use a subset of SOS commands, such as obtaining a stack trace. Some commands (like the ones related to garbage collection) will still require all committed pages, however.
For your production .NET-based applications, the best way to create mini dumps is with the old standby, ADPlus, formerly named Autodump+. The Debugging Tools for Windows package, freely downloadable at Debugging Tools for Windows, is the home for the latest version. For this column, I'll be using the October 2004 beta release 6.4.4.4 of the Debugging Tools for Windows. Every few months, there's a new release, so you should check back frequently to get the latest. (As an aside, if you are actively debugging with WinDBG or CDB/NTSD, you'll use the ".dump /ma <file>" command so that you can write an appropriate mini dump.)
ADPlus is actually a 5,000-line VBScript application that drives CDB, the console-based debugging UI on top of DBGENG.DLL. (By the way, CDB, NTSD, and WinDBG, are all different UIs on top of DBGENG.DLL so any commands I mention using one of the debuggers will work in all of the debuggers.) When you execute ADPlus, it's building up a script of debugger commands and pumping them into CDB, which does all the heavy lifting. For those of you who used ADPlus in the past, it was relatively nasty because you had to change values inside the actual VBScript file itself. Now ADPlus uses configuration files, in the ubiquitous XML format, to handle all the settings. That makes it much easier to use.
One of the big problems in production environments is that administrators get very nervous when you say you are going to install something on the server. The great news with ADPlus and its supporting files is that a simple XCOPY deployment works great. Just install the Debugging Tools for Windows on your development machine, as you'll need WinDBG to analyze the mini dumps, and then copy the installation directory to the server.
Since ADPlus uses VBScript, you'll need to have Windows Script Host (WSH) version 5.6 or later on the target machine. Run "CSCRIPT /?" at an MS-DOS® prompt to check the version installed. Another item you'll want to change is the default script host. You should use CSCRIPT.EXE instead of WSCRIPT.EXE so that output will go to Command windows instead of to message boxes, which can hang ADPlus until you click OK. Run "CSCRIPT /H:CScript" and you'll be all set.
In ADPlus, and the Debugging Tools for Windows overall, the documentation in DEBUGGER.CHM is outstanding. You definitely need to read through everything in there to get a good handle on how to run ADPlus as well as how to use the debugging tools. Make sure to concentrate on the Debugging Tools for Windows\Extra Tools\ADPlus in and Debugging Tools for Windows\Debugging Techniques. With the background out of the way, let's turn to actually using ADPlus.
ADPlus Usage
ADPlus has two usage modes: hang and crash. Hang mode is a bit of a misnomer; it really should be called "snap" because ADPlus instructs CDB to attach noninvasively to a process or processes, write out a mini dump, and detach. A noninvasive attach suspends the process and does not actually run under a debug loop so it works on all operating systems from Windows NT® 4.0 and above. If you're familiar with the old USERDUMP.EXE program, ADPlus is the complete replacement for it.
Crash mode, on the other hand, actually attaches the CDB debugger with a native debug loop so you can have the debugger respond to exceptions and even set breakpoints. For example, if you have a case where the ASP.NET worker process is shutting down unexpectedly, you can attach ADPlus in crash mode and have it set a breakpoint on KERNEL32.DLL's ExitProcess. When that breakpoint trips, you can write out a mini dump so you can walk the stack to see what started the process termination.
For most of your debugging, you'll be using ADPlus in hang mode so you can capture dumps at various intervals to see what's going on. While you can specify command-line options to ADPlus to handle all the work, I much prefer to use configuration files. That way I can get even more control over ADPlus and more information into the output.
Figure 1 shows the hang mode configuration file I like to use with .NET-based applications. The comments in the file indicate exactly what each option does and what commands I have selected to run. If you've ever played with WinDBG, you'll see that all the commands are familiar.
Figure 1 Hang Mode Configuration File
<!-- John Robbins - Bugslayer Column, MSDN Magazine --> <!-- Default HANG Options --> <ADPlus> <Settings> <!-- Set the mode to HANG --> <RunMode> HANG </RunMode> <!-- Snap the dumps, don't tell me about it --> <Option> Quiet </Option> </Settings> <HangActions> <!-- Clear out the default options --> <Option> Clear </Option> <!-- I like to have a dump (.dump /ma), --> <!-- callstacks (~*kb250), and loaded modules --> <!-- (lmv) in the log every time --> <Actions> FullDump; Stacks; LoadedModules </Actions> <!-- For custom actions, I want to see all --> <!-- the handle info, the managed CLR version, --> <!-- all managed threads, all managed --> <!-- call stacks, and the heap stats for --> <!-- objects/sums larger than 100 bytes. --> <!-- This assumes Debugging Tools for Windows --> <!-- 6.4.4.4 or later. --> <CustomActions> !handle 0 f; .load clr10\sos; !eeversion; !threads; ~*e!clrstack; !dh -stat -min 100 </CustomActions> </HangActions> </ADPlus>
To specify the configuration file on the ADPlus command line, use the –c option. Keep in mind that you'll need to specify the complete path to the configuration file if it is in any directory other than the same one containing ADPLUS.VBS. In addition to –c, you'll also need to specify the –o and –pn (or –p).
The output directory, set with the –o, tells ADPlus where to put the output. Each time ADPlus runs, it will create a directory in the output location with the format "Hang_Mode__<date>__Time__ <time>." In that directory, it will create the dump file, the log of the entire CDB output, and a file containing the running processes on the machine. Because ADPlus ensures a unique file name every time, you can run ADPlus repeatedly with the same command-line option and not lose previous runs.
The –pn option specifies the process name you want to attach and dump. For example, if you are using IIS 5.0, you'd specify "-pn ASPNET_WP.EXE" to grab your ASP.NET dumps. For IIS 6.0, you'd specify "-pn W3WP.EXE". By specifying an actual process name, if there are multiple copies of that process running, ADPlus will dump all of those processes. If you want to identify only a single process, you can do so by using the –p option, which will look for the process ID.
In the case of IIS 6.0, if you want to only snap a mini dump of a particular application pool, the command "TLIST.EXE –v" will show you all running processes and their command lines. (TLIST.EXE comes as part of the Debugging Tools for Windows.) Look through the list for the different W3WP.EXE instances and their –ap command-line options to identify the application pool in which each instance is running.
The majority of my mini dump creation has been performed with the hang mode option to ADPlus. The one problem is that you have to keep manually running the same command line repeatedly to get the runs over a period of time. I have included in this month's source code distribution a program called SpawnRepeatedly that I've used to automate the task of running ADPlus over longer periods of time. You can download it from the MSDN®Magazine Web site.
There's nothing too exciting about the program itself, but I've found it very useful to run on production systems and to snap a mini dump every couple of hours. Keep in mind that the ASP.NET worker process, or any process you attach ADPlus to in hang mode, suspends the process and could possibly cause problems. In most cases, the mini dump snap should take less than a minute, but for some servers, even that could cause problems so you may want to perform some test mini dumps first to gauge the impact.
Note that the configuration file shown in Figure 1 includes actions for some things that you can do later when analyzing the dumps. If you want to minimize the time you stop the application for those occasions where time is an issue, one alternative is to use a very lean configuration file, such as one where the only option used is Clear and where the only action used is FullDump.
The second mode for running ADPlus is directed at telling ADPlus to perform some operations when the process crashes. While in hang mode the debugger is simply suspending the process; in crash mode the debugger attaches using the Win32® native debugging API. Because of the excellent scriptability of the debugging engine, this gives you the opportunity to respond to specific exceptions, conditions, or when certain functions execute.
For example, the configuration file shown in Figure 2 snaps a mini dump whenever a first-chance exception occurs. Since .NET exceptions are implemented internally with Structured Exception Handling (SEH), each time an exception is thrown in your application, you'll have the exact dump that tells you where the exception occurs. I don't run this configuration on production systems because it can slow down the application too much, but it's invaluable in testing. Be aware that this configuration will take up a huge amount of disk space. While .NET makes exceptions much more palatable than they were in Win32, they are truly for exceptional conditions and you should not be seeing them thrown frequently in your normal operation.
Figure 2 Capturing a Mini Dump on Each Throw
<!-- John Robbins - Bugslayer Column, MSDN Magazine --> <!-- Default Crash Options --> <ADPlus> <Settings> <!-- Set the mode to CRASH --> <RunMode> CRASH </RunMode> <!-- Snap the dumps, don't tell me about it --> <Option> Quiet </Option> </Settings> <Exceptions> <!-- Do a unique mini dump with full heap on every --> <!-- first chance exception. --> <Config> <Code> AllExceptions </Code> <Actions1> FullDump; </Actions1> <ReturnAction1> GN </ReturnAction1> </Config> </Exceptions> </ADPlus>
Another crash mode configuration that's quite useful is telling ADPlus to snap a mini dump when a process calls ExitProcess. With ASP.NET, this configuration can help you determine why the worker process is jumping out of memory, which is probably the sign of some sort of COM Interop problem. The trick is to tell ADPlus to set the breakpoint on KERNEL32.DLL's ExitProcess and to do a mini dump with full heap at that point.
Figure 3 shows the ExitProcess configuration. By default, ADPlus wants to write out a mini dump on each and every first-chance exception (the moment of the throw). Figure 3 shows how to turn first-chance exception logging off.
Figure 3 ExitProcess Configuration
<!-- John Robbins - Bugslayer Column, MSDN Magazine --> <!-- Default Crash Options --> <ADPlus> <Settings> <!-- Set the mode to CRASH --> <RunMode> CRASH </RunMode> <!-- Snap the dumps, don't tell me about it --> <Option> Quiet </Option> </Settings> <Exceptions> <!-- Don't dump on first-chance exceptions --> <Option> NoDumpOnFirstChance </Option> </Exceptions> <Breakpoints> <NewBP> <!-- Set the breakpoint on ExitProcess --> <Address> kernel32!ExitProcess </Address> <!-- A normal breakpoint --> <Type> BP </Type> <!-- When hit, walk the stacks and do a --> <!-- mini dump with full heap. --> <Actions> FullDump; Stacks </Actions> <!-- After doing the actions, just --> <!-- continue --> <ReturnAction> G </ReturnAction> </NewBP> </Breakpoints> </ADPlus>
With dump creation under your belt, I want to show how you can take advantage of some of the new SOS features to get at that hard-to-find information.
What's New in SOS
If you read my previous column on SOS, you probably saw that SOS is good at dumping out information, but you have to spend your time manually grinding across it to find the really useful sections. The latest SOS, which is now part of WinDBG, makes finding key information, such as all the objects in the Generation 2 heap, trivial. Having spent far too many hours grinding through dumped output and manually trying to find what's in what generation, I'm thrilled to have these new capabilities.
The .NET Framework has the old copy of SOS, so to load the new SOS, you'll execute the ".load clr10\sos" command. The new version is targeted at common language runtime (CLR) 1.0 and 1.1 development. If you are playing with CLR 2.0, you'll want to load the SOS version that comes with the Framework.
The first thing you need to look at is the "!help" command because it looks like all the parameters to the information commands are now documented. You will also see that there are new commands specific to ASP.NET such as "!DumpASPNETCache" (also callable with the shortened "!dac") that will dump out all the data that is in the ASP.NET cache. Since the cache is a place where you are probably holding onto objects too long, being able to see the cache in one command is a huge improvement.
The command that's gotten the most attention is the one that's the most useful, "!DumpHeap" (also shortened to "!dh"). Not only is it now able to tell you which generation an object is in, but the most noticeable improvement is that it's many orders of magnitude faster. To see the statistics of all objects in the Generation 2 heap, the magic command is "!dh –stat –gen 2". That will produce comprehensive output like that shown in Figure 4, so you can see at a glance exactly what's sticking around too long. If you want to see the individual objects, remove the –stat argument and "!dh" dumps the actual objects themselves.
Figure 4 Statistics of Objects in the Generation 2 Heap
0:006> !dh -stat -gen 2 total 57,671 objects Generation 2 Statistics: MT Count TotalSize Class Name 0x79bf6b0c 1 12 System.IO.TextReader/NullTextReader 0x79bce950 1 12 System.Resources.FastResourceComparer 0x79bbba0c 1 12 System.Reflection.Cache.TypeNameCache 0x79bb6614 1 12 System.Security.PolicyManager 0x79bb5da8 1 12 System.Reflection.__Filters 0x79bad050 1 12 System.Security.SecurityRuntime 0x79bacb80 1 12 System.Security.CodeAccessSecurityEngine 0x79ba8a2c 1 12 System.__Filters 0x79ba8568 1 12 System.Reflection.Missing 0x79b9c8d0 1 12 System.IO.Stream/NullStream 0x0014eac8 1 12 Free 0x79c03b00 1 16 System.Security.Permissions.UIPermission 0x79bbbf14 1 16 System.Reflection.Cache.InternalCache 0x79bb9a78 1 16 System.Security.Policy.ZoneMembershipCondition 0x79bb6030 1 16 System.Security.Util.LocalSiteString 0x79bb52e4 1 16 System.Globalization.GlobalizationAssembly 0x79bab880 1 16 System.Enum/HashEntry 0x0099375c 1 16 System.Boolean[] 0x79bb3b74 1 20 System.Security.Util.DirectoryString 0x79b9fc8c 1 20 System.Text.UnicodeEncoding 0x79b9d374 1 20 System.Text.UTF8Encoding 0x79b98514 1 20 System.AppDomainSetup 0x79bce4a0 1 24 System.Globalization.TextInfo 0x79bbb4d8 2 24 System.Security.Permissions.StrongNamePublicKe... 0x79bb917c 1 24 System.LocalDataStoreMgr 0x79bac2ec 1 24 System.Security.PermissionTokenFactory 0x79b90e34 2 24 System.Object 0x79bd80d4 1 28 System.Security.Permissions.FileIOPermission 0x79bce824 1 28 System.Text.UTF8Encoding/UTF8Decoder 0x79bbc094 1 28 System.Reflection.Cache.ClearCacheHandler 0x009931d8 1 28 System.Reflection.Cache.InternalCacheItem[] 0x79bc9514 1 32 System.IO.__UnmanagedMemoryStream 0x79bb3544 1 32 System.Collections.ArrayList/SyncArrayList 0x79b95614 1 32 System.SharedStatics 0x79bef8ec 1 36 System.Security.Policy.UnionCodeGroup 0x79bcdaec 1 36 System.Resources.RuntimeResourceSet 0x79b9fed4 1 36 System.IO.BinaryReader 0x79bcf3f4 1 40 System.Runtime.Serialization.Formatters.Binary... 0x79bb4cdc 2 40 System.Globalization.CompareInfo 0x79bb2fcc 1 40 System.Security.Util.URLString 0x79bccfbc 1 44 System.Resources.ResourceManager 0x79bb55b4 1 44 System.Reflection.Module 0x79bb7800 2 48 System.Version 0x79bac190 3 48 System.Security.PermissionToken 0x79ba29d4 2 48 System.Reflection.Assembly 0x79b9f180 4 48 System.Security.Permissions.SecurityPermission 0x79bfb5a4 1 56 System.IO.StreamReader/NullStreamReader 0x79bb5eac 2 56 System.Reflection.TypeFilter 0x79baa5e0 1 56 System.Collections.Hashtable/SyncHashtable 0x79bb8384 1 60 System.Threading.Thread 0x79ba8cc8 5 60 System.Int32 0x79b94ee4 1 64 System.ExecutionEngineException 0x79b94dac 1 64 System.StackOverflowException 0x79b94c74 1 64 System.OutOfMemoryException 0x79bcd77c 1 68 System.Resources.ResourceReader 0x79b96e8c 1 80 System.AppDomain 0x79ba8b44 3 84 System.Reflection.MemberFilter 0x79baba4c 4 112 System.Security.Util.TokenBasedSet 0x79bb704c 3 132 System.Security.Policy.PolicyLevel 0x79b9e24c 5 140 System.Security.PermissionSet 0x009934a4 1 140 System.UInt64[] 0x79bba934 4 144 System.Security.NamedPermissionSet 0x79bb40e4 4 176 System.Globalization.CultureInfo 0x79bb7a94 2 240 System.Globalization.NumberFormatInfo 0x79bba56c 14 336 System.Security.Policy.StrongNameMembership... 0x009926b0 6 372 System.Int32[] 0x79ba7590 29 464 System.RuntimeType 0x79ba968c 12 624 System.Collections.Hashtable 0x0099236c 18 652 System.Char[] 0x00992970 12 1,872 System.Collections.Hashtable/bucket[] 0x79ba0884 113 3,164 System.Security.SecurityElement 0x79ba0d74 135 3,240 System.Collections.ArrayList 0x0099209c 149 6,072 System.Object[] 0x79ba13f4 413 6,608 System.Security.SecurityStringPair 0x00992f00 1 7,364 System.Reflection.Cache.TypeNameStruct[] 0x00992c3c 9 52,356 System.Byte[] 0x79b925c8 1,325 100,872 System.String Total 2,333 objects, Total size: 186,952 Large Objects Statistics: MT Count TotalSize Class Name Total 0 objects, Total size: 0
One undocumented feature of the "!dh" command is that you can easily look at the large object heap as well. In looking at a lot of output, I noticed some values listed as coming from Generation 3. Because the .NET garbage collector, at least according to everything I read, only has the three generations (0, 1, and 2), I was a little confused. Thinking it may be the large object heap, I manually dumped the large object heap and compared values. Sure enough, that's what I saw. To see the objects in the large object heap, use 3 as the generation like this: "!dh –stat –gen 3".
Another nifty new trick with "!dh" is the ability to specify a partial type string to search for. If you want to see just the statistics for Generation 2 on objects that have Security in the name, the command is: "!dh –stat –gen 2 –type Security". The new type option makes looking for your own objects trivial. If you haven't guessed this by now, you can mix and match command-line options to "!dh" all you want to see the heap in any way you want to slice and dice it. Finally, "!dh" works great on live debugging in addition to all those memory dumps you're about to create.
Following "!dh", the next most-used command, "!DumpObj" (also shortened to "!do"), has gotten a features boost as well. For those of you who have carpel tunnel syndrome from manually walking down object reference chains, the new –r # (recurse) switch will be much appreciated. If you want to look at an object and what it references, specify a number of objects you want to walk down and "!do" will take care of the rest. While it's a great feature, "!do" does not skip null/Nothing values. Instead, it tries to dump those objects to output, so you may see some errors in the minidump. If you see an error "Could not dump object, error returned was: 0x8007012b" just take a look at the previous CLASS value and you'll see that it is probably null/Nothing.
One of the previous biggest weaknesses of SOS was that it was impossible to look at an array. The new –v switch to "!do" now makes it trivial. Figure 5 shows a simple string array output. I've noticed that the –v switch will only dump out actual reference objects and will not display arrays with value elements.
Figure 5 Dumping an Array with SOS
0:000> !do -v 0x00b14500 Name: System.Object[] MethodTable 0x0098209c EEClass 0x00982018 Size 28(0x1c) bytes GC Generation: 0 Array: Rank 1, Type CLASS Element Type: System.Object Content: 3 items --- Will only dump out valid managed objects --- [ 0x00b14338 ] Name: System.String MethodTable 0x79b925c8 EEClass 0x79b92914 Size 40(0x28) bytes GC Generation: 0 mdToken: 0x0200000f (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Windows XP FieldDesc*: 0x79b92978 MT Field Offset Type Att Value Name 0x79b925c8 0x4000013 0x4 System.Int32 instance 11 m_arrayLength 0x79b925c8 0x4000014 0x8 System.Int32 instance 10 m_stringLength 0x79b925c8 0x4000015 0xc System.Char instance 0x57 m_firstChar 0x79b925c8 0x4000016 0 CLASS shared static Empty >> Domain:Value 0x0014aa70: 0x00a912b8 << 0x79b925c8 0x4000017 0x4 CLASS shared static WhitespaceChars >> Domain:Value 0x0014aa70: 0x00a912cc << [ 0x00b14360 ] Name: System.String MethodTable 0x79b925c8 EEClass 0x79b92914 Size 40(0x28) bytes GC Generation: 0 mdToken: 0x0200000f (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Server 2003 FieldDesc*: 0x79b92978 MT Field Offset Type Attr Value Name 0x79b925c8 0x4000013 0x4 System.Int32 instance 12 m_arrayLength 0x79b925c8 0x4000014 0x8 System.Int32 instance 11 m_stringLength 0x79b925c8 0x4000015 0xc System.Char instance 0x53 m_firstChar 0x79b925c8 0x4000016 CLASS shared static Empty >> Domain:Value 0x0014aa70: 0x00a912b8 << 0x79b925c8 0x4000017 0x4 CLASS shared static WhitespaceChars >> Domain:Value 0x0014aa70: 0x00a912cc << [ 0x00b14388 ] Name: System.String MethodTable 0x79b925c8 EEClass 0x79b92914 Size 44(0x2c) bytes GC Generation: 0 mdToken: 0x0200000f (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll) String: Windows 2000 FieldDesc*: 0x79b92978 MT Field Offset Type Attr Value Name 0x79b925c8 0x4000013 0x4 System.Int32 instance 13 m_arrayLength 0x79b925c8 0x4000014 0x8 System.Int32 instance 12 m_stringLength 0x79b925c8 0x4000015 0xc System.Char instance 0x57 m_firstChar 0x79b925c8 0x4000016 0 CLASS shared static Empty >> Domain:Value 0x0014aa70: 0x00a912b8 << 0x79b925c8 0x4000017 0x4 CLASS shared static WhitespaceChars >> Domain:Value 0x0014aa70: 0x00a912cc <<
In addition to poor array handling, in the past SOS also had problems with collections, unless you had a lot of time to manually dump them with "!do". The new "!DumpCollection" (short version: "!dc") will display any collection that's derived from the ICollection interface, which means nearly everything. In the 6.4.4.4 beta release of WinDBG, the "!dc" command is not working too well, but it will be fixed in the final release.
One extremely tough problem to track down in mixed-mode applications is a GCHandle leak. The System.Runtime.InteropServices.GCHandle structure is what's used to hold onto a piece of managed memory that's passed to native code. For the most part, Interop works like a dream, but there are times when the native code no longer references the memory, and the managed side wasn't notified. At such times you have this GCHandle structure keeping your managed objects alive, and it will never be freed. This is especially nasty in the case of pinned memory since the garbage collector will have to move everything else around that location. As you can surmise, the coordination between your managed and native sides is extremely important.
To keep an eye on the GCHandle structures in use, the new "!gchandles" command reports the handle statistics and those managed objects which are being held in the GCHandles table. An example output of the command is in Figure 6. The important numbers are the Strong and Pinned Handles. Strong handles mean the memory is referenced and cannot be garbage collected. Pinned means the object is referenced and is locked in memory. Keep in mind that strong and pinned memory happens to parameters implicitly when you make COM and P/Invoke calls. You'll want to keep an eye on those two numbers and if you see them creeping up, you've got a problem.
Figure 6 Output from the !GCHandles Command
0:000> !gchandles Scan HandleTable 0x168320 Scan HandleTable 0x14b4b8 GC Handle Statistics: Strong Handles: 33 Pinned Handles: 0 Async Pinned Handles: 0 Ref Count Handles: 0 Weak Long Handles: 0 Weak Short Handles: 1 Other Handles: 0 Statistics: MT Count TotalSize Class Name 0x79b925c8 1 20 System.String 0x79c05374 1 24 System.IO.TextWriter/SyncTextWriter 0x79b95614 1 32 System.SharedStatics 0x79bb55b4 1 44 System.Reflection.Module 0x79bad2dc 2 48 System.Security.FrameSecurityDescriptor 0x79b94ee4 1 64 System.ExecutionEngineException 0x79b94dac 1 64 System.StackOverflowException 0x79b94c74 1 64 System.OutOfMemoryException 0x79b96e8c 1 80 System.AppDomain 0x79bb8384 2 120 System.Threading.Thread 0x79ba29d4 5 120 System.Reflection.Assembly 0x79b9e24c 10 280 System.Security.PermissionSet 0x0098209c 4 12,320 System.Object[] 0x00982c3c 3 73,652 System.Byte[] Total 34 objects, Total size: 86,932
In addition to "!GCHandles", the new "!GCHandleLeaks" will help you track down those leaks. "!GCHandleLeaks" works by looking at the handle values in the GCHandle table and scanning all of your process memory looking for references to those objects. If it finds a reference in memory that indicates the memory is still being used, then it's not a leak. If there's no reference to the object the GCHandle structure wraps, the command reports it as a leak. There's a small possibility that there could be false positive results if the native code is holding the managed pointer in a nonstandard way such as a mask value.
The end of the "!GCHandleLeaks" output will show the handles that it believes are leaked. Those values are the actual GCHandle structures. If you take a look at the GCHandle structure with Lutz Roeder's .NET Reflector, you'll see that the structure only contains an int value. To see the actual managed object leaked, perform a debugger DD (dump DWORD) command on the value that was reported leaked; that will show the object address so you can execute the "!do" command to see it.
Another new command, "!DumpAllExceptions" (also shortened to "!dae"), can be used whenever you want to see any exceptions that are still in the GC Heap.
The final new command I want to mention is the awesome "!SaveModule" command. If you're in a situation where you're debugging (either live or a mini dump), and you need to see the source code for a module, it can sometimes be difficult. However, for a managed module, you can pass the load address of the assembly and an output name to the "!SaveModule" like this: "!SaveModule 00400000 temp.exe". That will write out the complete assembly to disk so you can use Reflector to decompile the binary! Why suffer through trying to find the right version of the source code when you've got the actual source right here?
Wrap-Up
In this column, I hope I've given you some of the tips and tricks of gathering the best mini dumps possible in a production environment. You have the ADPlus configuration files I use when debugging so you can use them as a base to start building the custom configurations you'll need for your applications. While the ADPlus documentation is quite good, I strongly recommend that you look at the actual ADPlus source code so you can figure out exactly what will happen with various options.
Finally, I'm sure you're going to love the new SOS. The fact that you can easily see the objects in a specific generation makes it massively more useful that it used to be. While it's still not as easy as debugging in the Visual Studio® .NET debugger, it is the only tool I consider for tackling those nasty production-only problems.
Tips
Tip 67 Since I'm a tools kind of guy, I would be remiss if I also didn't plug Jonathan de Halleux's outstanding MBUnit. It's fully compatible with NUnit, but offers advanced features for creating great tests such as testing against different cultures, data driven testing, and process testing. MBUnit is replacing NUnit for me.
Tip 68 While MBUnit is excellent, the other testing tool you need is from Jamie Cansdale: TestDriven.NET. It's a phenomenal Visual Studio .NET Add-In that allows you to plug in tools like MBUnit, NUnit, and even the upcoming Microsoft Team System into a single one-click testing environment. All I can say is wow!
Send your questions and comments for John to slayer@microsoft.com.
John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for .NET and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at www.wintellect.com.