Bugslayer
SUPERASSERT Goes .NET
John Robbins
Code download available at:Bugslayer0511.exe(221 KB)
Contents
Requirements
Using SUPERASSERT.NET
Implementation Details
End Notes
Those of you who have been reading this old Bugslayer column over the last nine years have branded into your frontal lobe a single word: ASSERT! Anytime you can have the code tell you about a problem instead of having to find it by slaving away with a debugger is a huge timesaver. As you'll remember from your native C++ days, you had to do all your own proactive programming from scratch. In the Microsoft® .NET Framework world, the heavy lifting is done by the Base Class Library's (BCL) Debug.Assert method. Add to that the outstanding extensibility in the TraceListeners and you've got a debugging system that's ready to handle just about anything.
Even though there's such great debugging support, it's not perfect. When a .NET assertion is triggered, you can't see the parameters in the stack trace and you can't look at the call stacks of other threads in the process. But when you consider the sandboxing and security that .NET offers, you can see why the system hides that information. If you could get parameters and call stacks from other threads, you'd be right back in the wild days of native C++.
That said, there are still times when you really need to see those other call stacks and parameters. Think about the times you can't duplicate a bug on your development machine and you need to be running on a test lab machine. If you could look at the data and snap a mini dump of the app at assertion time, you'd almost give up your 1993 PDC Slinky or something equally significant.
What's needed for .NET is the equivalent of the native C++ SUPERASSERT that I wrote way back in 1999 when this magazine was Microsoft Systems Journal (see Bugslayer - February 1999), and extended in my book Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press®, 2003). Not only did SUPERASSERT walk the stack for any thread (along with all parameters and locals), it had mini dump support, extra cool ignore options, and even allowed you to e-mail the assertion. SUPERASSERT was supposed to be the last assertion code you would ever need.
Many of you have asked if it would be possible to do SUPERASSERT in the managed world. I had my doubts, but I've come up with something close, which I'll present here.
Requirements
Let's begin with the requirements. I needed to be able to walk the stack for other threads in the process and write a mini dump when the assertion UI was shown—some pretty serious requirements.
I wanted something very similar to the user interface from the native SUPERASSERT I'd done for my book because it has worked well. I know that testers and other developers need a way to mark an assertion as ignored so it does not keep popping up. For example, if you've got normal error handling reporting to the user that the disk is full, having testers and developers fight through three or four assertion displays to get to the normal error is extremely annoying. Plus, they may need to turn off all assertions for automated tests.
Another key requirement is the ability to copy out of the assertion UI to paste into a bug report. Equally important is the option to e-mail the assertion to the developer who is responsible.
The ability to jump into the debugger when necessary is also critical. Debugging in .NET means debugging managed and native code. Visual Studio® .NET is more than sufficient for most of your managed debugging needs, but if you're using native DLLs or COM components, or have to see odd managed memory problems, you need the native debugging power of tools such as Son of Strike (SOS), which I've talked about before (see SOS: It's Not Just an ABBA Song Anymore and Mini Dump Snapshots and the New SOS). One requirement I had in the native C++ version of SUPERASSERT was the option to suspend all other threads in the process from the asserting thread. That allowed you to avoid synchronization issues whenever an assertion triggered. While it's possible to suspend threads in the .NET version, there's the small problem of garbage and finalizer threads. If I would suspend threads, I'd kill the memory management system in .NET and that's certainly not a good idea.
Finally, I wanted the code to work on both Win32® and Win64, but be able to run as a 32-bit application on Win32 and a 64-bit application on Win64.
Using SUPERASSERT.NET
When you're talking about using Debug.Assert and creating new output, you're really looking at a custom TraceListener. In fact, SuperAssertTraceListener is derived from DefaultTraceListener, so I only had to override the Fail methods.
It's pretty simple to meet all my requirements for free (almost). First, copy to your application's location the three DLLs that make up the utility (Bugslayer.DLL, BugslayerUtil.SuperAssertSettings.dll, and BugslayerUtil.SuperAssertSettings.XmlSerializers.dll). Next, add SuperAssertTraceListener from Bugslayer.DLL to the Debug.Listeners collection. You can do that either programmatically or by using TraceListeners in your app.config.
This last step is a little more interesting and you'll see why this is required when I talk about the implementation. You'll need to install the Debugging Tools for Windows on at least one machine. For test machines, or production if you are so inclined, you just need to XCOPY the Debugging Tools for Windows directory to the machine.
After you have an assertion in your application, you'll see the assertion message in Figure 1. The edit control at the top of the window shows the message and detailed message parameters you passed to Debug.Assert. You also see the module, source, and line where the assertion occurred. So far, it's the same information you'd see in the standard DefaultTraceListener.
Figure 1** SUPERASSERT .NET Main Dialog **
The LastError value shows you the last native Windows error as reported by GetLastError, which can be helpful, especially if you're doing lots of interop. Note that the value displayed here might not have anything to do with the assertion you're seeing. My code saves off the last error value as soon as it's called. However, the last error value could have been changed by a previous TraceListener in the Listener collection.
The penultimate value in the edit control is the number of times this particular Debug.Assert failed. The last value in the edit control is the number of native Window handles your process currently has open. Leaking handles is a huge problem in both native and managed code so seeing the total number of handles can, in fact, help you detect potential problems.
I'm sure you can guess what the Ignore Once button does for the current assertion. The Abort Program button is a true death button, as it will call the Windows TerminateProcess function to rip the process completly out of memory. You'll want to be extra careful pressing that button as there's no turning back.
The Managed Debugger button triggers the managed debugger. If you are debugging the process, it will call Debugger.Break to stop in the debugger. If there is no debugger present, SUPERASSERT.NET will call Debugger.Launch to start the just-in-time debugger process so you can choose the debugger to use. As you would expect, you need to have sufficient operating system privileges to debug the process.
The Copy To Clipboard button copies the values from all textboxes in the dialog to the clipboard. Don't be surprised if the button is disabled in your application. It turns out that the .NET clipboard class relies on COM, so if your main thread is not marked as a single-threaded apartment (STA) thread, you'll get interesting exceptions attempting to use the clipboard classes. There are a few solutions to this problem. One is to spin up a new STA thread and do all of the clipboard interaction on that thread. Another is to use interop to access the Windows clipboard API. In a future release, I'll do the latter.
The Create Minidump button brings up a simple dialog where you can specify the name of the mini dump you want to write. As with the clipboard classes, the .NET FileBrowserDialog class uses COM so it's not safe to use without an STA main thread and, as a result, the dialog you'll see is nothing fancy. However, the dialog will make sure the path to the file exists and give you a simple check option to automatically put the date and timestamp on the file name to ensure the mini dump file is unique.
The mini dumps that I created are the appropriate full-memory mini dumps, which means you can feel free to bang away at them with SOS and really see what's going on in your application. That also means they can get huge. For the simple test program, I created full-memory mini dumps on Windows XP Professional x64 Edition that were 421MB!
The e-mail assertion will only be active if you've made a very small change to your code. In order to identify who owns the code, you'll need to add the CodeOwner attribute from the Bugslayer.Diagnostics namespace in BugslayerUtil.DLL to your classes, as shown in the following snippet:
[CodeOwner ( "John Robbins" , "john@wintellect.com" )] class Program ...
SUPERASSERT.NET will look up the stack of the assertion for the first class with the CodeOwner attribute and use that e-mail address as the To field for the mail. When using the CodeOwner attribute, please make sure to remove my e-mail address because, as you can imagine, I just don't need any more mail. The e-mail code in SUPERASSERT.NET uses the new SmtpClient class from the System.Net.Mail namespace.
When you press the Email Assertion button, you'll see the window in Figure 2, which has the text from the main window in the body portion and the To and Subject fields already set. The default operation is to use the mailSettings element in the app.config or machine.config. If you're working on an application where your e-mailed assertions will be coming from inside your firewall, you can set up a hardcoded account with user names and passwords in the app.config so that e-mailing assertions is trivial.
Figure 2** E-Mail the Assertion **
If you're working on an application in which people outside your firewall will be sending messages, you'll want to tell them to deselect the Use default SMTP credentials from app.config. When they uncheck that box, the SMTP Settings button is enabled and they can enter in their username, host, port, and secure sockets layer (SSL) settings for their SMPT server. SUPERASSERT.NET saves those settings so the user doesn't have to enter them again. By the way, all SUPERASSERT.NET settings are being stored in C:\Documents and Settings\<user>\Local Settings\Application Data\Bugslayer\SuperAssertTraceListener\1.0.0.0\Settings.Config so they are global to all processes on the machine.
When users set their own SMTP settings, they'll be prompted for their password. The SUPERASSERT.NET code does not store that password, but it does appear in unencrypted form in memory briefly, so it may be a security violation depending on your company's security standards.
Overall, I'm not so happy with the SMTP approach for e-mailing assertions because it's not as seamless as it should be. Maybe in SUPERASSERT.NET 2.0, I'll write a little plug-in architecture that allows you to use the e-mail program on the user's machine so it's a better experience. Note that if you're using Beta 2, the SmtpClient code does not work with Google's GMail as your SMTP server. This is expected to be fixed in the final release.
The More>> button in Figure 1 is pretty exciting. When you click on it, the dialog expands to look like Figure 3. Since I don't do much UI development, I expect you to shout ooh and aah every time you click on the More>> and Less<< buttons. It's also fun to click on those buttons very fast.
Figure 3** Expanded SUPERASSERT.NET **
The Ignore group contains advanced options for disabling specific assertions or even all assertions, and should be used with care. The Stack group allows you to walk the stacks of all the other managed threads in the application. The default is to walk only the managed stack that had the assertion. The Thread ID combobox allows you to look at the call stacks from other threads. As a byproduct of the method I'm using to get the other managed thread stacks, you won't see the source file name and line for the other threads, only the thread with the assertion.
The last group box, Native Debuggers for SOS, lets you choose either WinDBG or CDB (Console Debugger) to do the hardcore, ninja-level debugging. When SUPERASSERT.NET spawns the debuggers it automatically loads the appropriate version of SOS so you can immediately start hacking at the managed heap or stack with no slowdowns.
Figure 4** Options Dialog **
In past columns, I've discussed WinDBG, but you may have never heard of CDB. Both debuggers are essentially the same. WinDBG has a GUI (and I use that term very loosely) and CDB is console-based. The Command window in WinDBG and CDB itself are just wrappers on the same DLL—DBGENG.DLL.
The last piece of the UI I'll mention is the Options dialog, which is accessible through the Options menu item on the system menu and is shown in Figure 4. You can figure out what the options do just by looking at the dialog, which is set to the defaults. You'll probably want to leave the stack-walking set to initially show only the thread that had the assertion because walking the other threads is slow.
Also, SUPERASSERT.NET is smart about its z-order. If you're running under any flavor of debugger, it knows not to set the TopMost property because doing that will make the SUPERASSERT.NET window appear on top of the debugger, making debugging impossible. Any options you set in this dialog are also saved to the settings file I mentioned earlier.
Implementation Details
Implementing SUPERASSERT.NET was what you might call "fun," if you're so inclined. Although I wasn't able to get the parameters and locals of items on the stack directly in the SUPERASSERT.NET dialog, I was able to get a glance at other thread stacks from an assertion and write correct mini dumps that can be debugged at a later time.
On any project, I keep a list of problems I expect to run into before I start coding. An important problem with SUPERASSERT.NET was how to ignore and skip individual assertions. From a StackTrace object, you can access the individual items on the class with their StackFrame class. The stack frame has the Type, which gives you the fully qualified name of the class, the method name, and the intermediate language (IL) offset into the method. Combine those three items and you've got a unique hash key to associate with an object that contains fail and ignore counts.
The number-one item on my tough problems list was how to walk the other threads and create a correct mini dump. I'd noticed in the .NET Framework 2.0 documentation that the StackTrace class has a new constructor that takes a Thread object. The idea is that you can pass in any Thread object and get that thread's stack. That gave me some hope of enumerating the threads from inside the application and getting their corresponding Thread objects.
You may have noticed that the Process.Threads property is a ProcessThreadCollection that contains the ProcessThread objects for all the native threads in the process. Thinking that there might be a way to convert those ProcessThread objects into the equivalent Thread object, I spent some serious quality time with the compiler and Reflector trying and looking at ideas. After pounding through everything I could think of, I realized that the only way I was going to get the information from other stacks was to spawn off a process that would pass that information back into SUPERASSERT.NET. I pulled up the common language runtime (CLR) Debugging API headers and suddenly realized that I didn't need to write my own debugger to get the stacks, I already had one on the computer, CDB.EXE paired with a little SOS pixie dust in the form of ~*e!clrstack.
The ~*e command in CDB/WinDBG says to run the following for all threads in the process. The command, !clrstack, is the SOS command to walk the managed stack. All I needed to do was spawn off CDB, turn on logging in CDB, run the ~*e!clrstack, close the log, and parse the results. While the idea would be to walk all the stacks from inside the process, there's no viable way to make that happen. This is a fair trade-off, especially considering that you just need to XCOPY the Debugging Tools for Windows directory to the target machine. While I haven't tried it, you may be able to cherry pick binaries out of the Debugging Tools for Windows directory and still have SUPERASSERT.NET run perfectly for you.
Some of you might be wondering if using CDB like this will work because of processes that are not running with administrator privileges or the SeDebugPrivilege. The SeDebugPrivilege is important only if you are going across user accounts on the machine. The good news is that SUPERASSERT.NET is spawning the CDB process on itself, so as long as the process has permissions to see and run CDB on the disk, this approach will work.
As soon as I tried my idea, I of course ran into a problem. I was starting CDB with the -c command line, which allows you to pass the commands to execute when the debugger starts. I was passing in a string with multiple commands separated by the standard semicolons to indicate CR/LF. Unfortunately, CDB randomly skips some of the commands in the string. One run would have the log file opened and the next would not. I switched to putting the commands in a text file, with one command per line and using the —cf option to tell CDB to read the commands out of the script file. That worked so I could get the call stacks.
The script format string contains the following CDB commands:
.loadby sos mscorwks ~*e!clrstack .logopen "{0}" ~*e!clrstack q
The first line tells CDB to load the SOS.DLL file that's in the same directory as the core CLR file, MSCORWKS.DLL. The second line walks all the stacks. You see two stack walks in the script. The reason is that to do the stack walking, !clrstack forces appropriate PDB source files to load, and CDB (being the output spew monster) interjects lots of messages about what loaded and didn't. Because those messages are random, parsing is drastically harder, so I just let !clrstack run and CDB can spew all it wants there. After opening the log file, which is actually in the temp directory, I do the real ~*e!clrstack that's just the output from SOS.
The command-line format string I pass to CDB.EXE is: -p {0} -pv -cf "{1}". The -p is the process ID to attach. The -pv is a neat trick in that it tells CDB.EXE to do a noninvasive attach. That way you are not actually debugging the application. On Windows 2000, if CDB, which is a native-only debugger, attached normally to the process, as soon as the debugger ended, the .NET process that spawned CDB.EXE would end because there's no support for native detaching. The —cf is the script file to run. Like the output file, the script file is written to the temporary directory.
As soon as CDB ends, I pick up the output file and parse out just the managed stacks using the following regular expression string:
const String regExtractClrStackPattern = @"(?# Get the thread ID for this block into the TID group)"+ @"(Id:\s(?<TID>0x[0-9a-fA-F]+)\s\([0-9]+\)\r\n+)"+ @"(?# Skip over the header pieces for x64 or x86)"+ @"(?:^(Child|ESP).+\r\n)"+ @"(?# Grab the stack into the CALLSTACK group)"+ @"(?<CALLSTACK>(^[0-9a-fA-F]+\s[0-9a-fA-F]+\s.+\r\n)+)";
You can read the comments embedded in the expression itself to see what's going on.
After pulling out just the managed call stacks from the output, I remove the stack pointer and return addresses from the strings. After that, it's just a matter of storing those call stack strings so I can show them when requested by the user.
One set of data I wanted to add to the call stacks was parameter and local values. You might be wondering why I didn't take advantage of the -a option to !clrstack, which dumps locals and parameters. With the .NET Framework 1.1 versions of SOS.DLL, the output is good. Unfortunately, with the .NET Framework 2.0 Beta 2, the version of SOS.DLL I'm using has some problems with locals and parameters (it shows some, but not others). This problem is supposed to be fixed by the release of the .NET Framework 2.0. I'll update SUPERASSERT.NET to take advantage of the fixed version of SOS.DLL. Since I'm using CDB to do the stack walking, you can probably guess how I'm handling the mini dump creation from SUPERASSERT.NET.
End Notes
For those of you who were hoping to see some serious work to get consistent correct call stacks from other threads from inside a managed application, I'm sorry to disappoint. While it may be doable, it would be extremely fragile and hard to fully support on multiple platforms. Overall, I'm happy with the approach of having CDB take care of getting stacks and mini dumps because it works and it's maintainable.
The approach I took also produces SUPERASSERT.NET, which in the end is the most important outcome. Now you can get an idea of the state of your application during an assertion. Better than that, you can get the mini dump for later and even get a native debugger with SOS all over the process when you want as well. The idea is anything that can help you debug faster is a great thing!
Tip 71 SUPERASSERT.NET is a TraceListener. Another fantastic example of a TraceListener is Josh Einstein's TerminalTraceListener. TerminalTraceListener opens a port and allows you to connect to your app through Telnet so you can watch your traces in the lab or even across the globe. I wish I would have thought of TerminalTraceListener!
Figure 5** Keyboard File **
Tip 72 Peter Provost showed me an amazing undocumented trick in Visual Studio .NET for all the total keyboard people out there. If you go to the find combobox (Ctrl+D with the default keyboard mapping) and you type "> open" (without quotes). The ">" flips the find combobox into a mini Command window. The "open" part is the aliased command to the "file open" action. The super part is when you press space and the first character of the file you want to open anywhere in your solution, a dropdown window pops up with the files and folders that start with that letter. See Figure 5 for the magic.
Send your questions and comments for John to slayer@microsoft.com.
John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in .NET and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at www.wintellect.com.