Here is the same content in en-us.
This article may contain URLs that were valid when originally published, but now link to sites or pages that no longer exist. To maintain the flow of the article, we've left these URLs in the text, but disabled the links.
|Tester, Take Two-TESTREC.EXE Updates Previous Version of the Tester Utility |
|John Robbins |
Code for this article: Bugslayer0600.exe (256KB)
| ecently, someone asked me to talk about a time during the development process when I learn the most. Being a debugger kind of guy, I'm sure they expected me to talk about the times I've worked on the nastiest bugs. While you do learn a bit slogging through the debugger, the real eye-opening learning experiences have occurred when I have had to do a subsequent version of a project.|
When you create the new release, the first thing you have to do, which can be quite instructive, is deal with all the bugs that you left in the product or that surfaced in the field. And sometimes you have to face the fact that what you thought was a wonderful design that would scale infinitely into the future just plain stinks! To me, that epiphany is a true Zen moment. You have to bravely face the mortality of something you created and, most importantly, you have to make it right. What makes the situation even more painful is that you have to admit to your teammates and to your manager that you weren't as smart as you thought you were. While your ego takes a beating, that's the price you pay for enlightenment.
On the other hand, in creating a subsequent version you can sometimes reach true nirvana and still learn a great deal. Some of my proudest moments have been when I learned that my design was sufficiently flexible and strong enough to extend in ways that I had never originally thought of. While that's wonderful when you are doing the subsequent version work yourself, the ultimate high is when others can pick up your work and easily take it in new directions. That's the moment where the "guru" term starts getting flung around!
To get the good and the bad, you need to live with your designs for a while. However, that doesn't mean you should live with the mistakes forever. If a design or implementation will not meet your needs for the future, you should consider scheduling time to rewrite the problem area. Whenever I am scheduling a project, I always expect to spend 15-25 percent of the time as reinvestment in the code base. Sometimes the problem areas are obvious, but other times they are not. To find the less obvious areas, I spelunk through the bug tracking system looking for the modules or subsystems that had the highest number of problems.
So far I have been talking mostly about implementation problems. But a much more difficult determination to make is whether you got your feature set wrong. If you are working on very vertical applications, sometimes you are the only game in town, so your customers don't know any better. Most of us are in competitive situations and can tell by looking at our company's bottom line if we hit the mark. While you might think you have the coolest product since sliced bread, your customers vote with their dollars. If it's too hard for them to get their job done, they might vote for your competitor's product. Please note that I am talking about missing features and functionality, not plain buggy products. I'm assuming you have been reading the ol' Bugslayer column all along and don't suffer from those afflictions any more.
Let's take a look at an app that's missing features or functionality. Back in the April 1999 issue of MicrosoftÂ® Systems Journal, I showed you a cool utility called Tester. Tester allows you to send keystrokes to another application so you can easily develop regression tests that take care of testing for you. Since it is a COM-based system, you can use any programming language you want to write the tests. Unfortunately, it's far too difficult to write those tests. Even I found it harder to use than it should have been.
In the original column, I mentioned that I needed to break down Tester into two parts: the part that plays back keystrokes and the part that records keystrokes. I meant to write the second half earlier, but other column topics hijacked my attention and sent me off on interesting tangents. After a couple of e-mails telling me that I didn't do the full jobâ"meaning I was missing featuresâ"I thought I had better take care of business and make a more feature-complete utility. I had my Zen moment.
This month, I will finish the second part of Tester with TESTREC.EXE, the recorder utility. With this utility, Tester as a whole is much easier to use and will allow you to whip up some automated regression tests that help speed up your testing considerably. I'll call the new version Tester Two.
First, I will quickly reintroduce Tester so everyone has an understanding of the first version's capabilities. Then I'll move into the requirements for TESTREC.EXE. Finally, I will discuss some of the implementation highlights. As you'll see, the implementation was much more than I bargained for.
Tester Recap Tester is a COM object that offers a couple of interfaces that allow you to get information about and control other windows on a computer. The following were my requirements when building the first version:
I personally like to use WindowsÂ® Script Host (WSH) as my scripting environment. Figure 1 shows an example script, which starts Notepad and says a few things. While playing the keys and fiddling with windows are concepts that are easy to grasp, handling arbitrary window creation and destruction is a little more elusive. It's a very nice feature, so I want to make sure that everyone understands it.
- Given an input string of keystrokes formatted the way they would be if you were going to pass them to the Visual BasicÂ® SendKeys function, Tester can play the keystrokes to the active window.
- Tester can find any top-level or child window by title or class.
- Given any arbitrary HWND, Tester can get all the properties of the window.
- Tester must notify the user's script of specific window creation or destruction so it can handle potential error conditions or do advanced window handling.
One of the most difficult issues you face when automating an application is when the unexpected happens. If you are doing what you should be doing, you will be using Tester scripts on your debug builds. Since you are also using liberal assertions, you never know when one can pop up. Obviously, you don't want to have to write your script so that you have to check for those assertion message boxes each time you do a particular operation in your script. By having the notification handlers, you can set a handler in the background that is triggered by Tester when a window is created or destroyed and matches certain criteria. That way you have an extra level of safety in your scripts to make sure you are automating the correct windows properly and not accidentally telling Windows Explorer to delete your application's directory.
Tester Two Requirements The main point of Tester is to send keystrokes over to another window. Tester does that quite well, but the problem is the format of the TInput PlayKeys method. I used the Visual Basic SendKeys format, which is not intuitive at all. Here are the requirements that I laid out for Tester Two:
In general, the requirements seem straightforward. However, you are probably wondering about recording and playing back mouse movements and clicks. Originally, I had planned to include them, but I ran into so many problems getting the keyboard recording straight that I simply ran out of time. While I could have simply done something with hardcoded pixels, I desperately wanted to find a way to avoid scripts that were hardcoded to the resolution on which you recorded them. To me, a regression testing script with a hardcoded screen resolution is not very useful. Consequently, mouse handling remains a goal for a future version of Tester.
- The ability to record keystrokes and place them into a string compatible with PlayKeys and SendKeys.
- When generating a Tester script, the script is self-contained so the saved script is ready to run.
- The user can edit the automatically generated script before saving it.
- Tester can properly set focus to a specific window, including any child control, to ensure that playback goes to the correct window.
Using Tester Two Recording your testing scripts is a piece of cake. Start TESTREC.EXE, press CTRL+R or click on the record button in the toolbar, press ALT+TAB to shift to your application, and type away. When you start recording, TESTREC.EXE minimizes itself to the task bar and changes its title to RECORDING! so you know what it's doing.
After recording the keystrokes you want, there are several ways to stop recording. The simplest way is to set the focus to the TESTREC.EXE application by clicking on it or using ALT+TAB to set the focus. Alternatively, pressing CTRL+BREAK or CTRL+ALT+DEL will stop recording as well. When you are recording, the edit control in TESTREC.EXE will contain a VBScript-compatible script of your actions. You can edit the script or save it to a file.
After you have recorded and saved a script, just run it under WSH to play back the keystrokes. The Tester COM object in TESTER.DLL uses TINPUTHLP.DLL and TNOTIFYHLP.DLL, so make sure they are in your path.
Before I move on to the implementation, I want to give you a few tips on how to maximize your Tester usage. First, like any development task, you should have a plan for what you intend to accomplish before you record in one session someone typing all of Shakespeare's sonnets into your application.
Your goal should be to make your scripts as generalized as possible so you can reuse them. For example, you should only have one script to open files. This means you should only record small operations. You can chain a bunch of small scripts together with .WSF files for WSH 2.0 in Windows 2000, or with batch files if you have to support Windows NTÂ® 4.0 or Windows 98.
After you record the keystrokes to your application, you might want to edit the output so that it appears more readable. TESTREC.EXE just records the keystrokes as a stream, with predefined breaks for outputting the information, so you should tighten up the calls to PlayKeys. You should also edit the recorded keys to remove any mistakes (such as BACKSPACE) so you are only playing back known, good input.
TESTREC.EXE knows how to ensure that the proper window gets focus. In Tester Two, this is done through the ALT+TAB handling, so you might want to ALT+TAB to your application every once in a while without actually shifting focus to a new application. For example, if you are testing a data entry application, enter the data and tab to the next field. Press ALT+TAB to generate the appropriate code to set the focus to the new child window. Part of setting the specific focus involves verifying that the child window exists. The extra ALT+TAB keystrokes will ensure that the appropriate child window exists and is ready for input.
Implementing TESTREC.EXE The TESTREC.EXE implementation turned out to be much more involved than I ever expected. The issue wasn't having to write tons of code, but trying to get the proper states set when building up the output keystrokes. Making the output understandable to humans was a challenge as well.
The first thing I needed to figure out was how to record keystrokes. In Windows, there is only one way: a journal record hook. There's nothing very exciting about journal hooks except handling the WM_CANCELJOURNAL message properly. When the user presses CTRL+ALT+DEL, the operating system executes any active journal recording hooks. This makes sense because it would be a pretty serious security breach to allow an application to record the keystrokes that make up the user's password. To handle WM_CANCELJOURNAL in a manner that keeps the implementation details hidden, I used a message filter to monitor for it coming through. You can see all the hook details in HOOKCODE.CPP in this month's code distribution at the link at the top of this article.
Of course, since a journal record hook is a global hook, debugging gets to be a little interesting. I played around to see if I could debug everything on one machine; on Windows NT 4.0, I proceeded to lock up the entire user interface. The only way to debug TESTREC.EXE is by remote debugging with the Visual C++Â® debugger. For more information about remote debugging, see my Bugslayer column in the August 1999 issue of MSJ.
I did learn something new about the Visual C++ debugger while working on TESTREC.EXE. When you have remote debugging set up, the Build Execute program name menu option will start the program on the remote machine. I thought that was pretty cool!
Before I jump into the really nasty guts of keystroke generation, you might want to look for SendKeys on MSDNâ¢ Online. The discussion that follows assumes you are familiar with the basic gyrations of the keystroke format.
After I was merrily recording with a shell of a hook, I needed to think a bit about the best approach for implementing the recording processing. I decided to keep the actual recording and output parts generic so I could change them at will as well as test them in isolation. Since journal record hooks can reside inside the EXE and any hook processing should be as lightweight as possible, I simply used a C++ class, CRecordingEngine, to handle the processing. The CRecordingEngine class constructor takes a pointer to a CRecordingUIOutput class. This class, which I defined in RECORDINGENGINE.H along with CRecordingEngine, is an abstract base class that CRecordingEngine calls for output.
Figure 2 shows RECORDINGENGINE.CPP, the part where I spent way too much time. I had to do horrendous amounts of testing to make sure I had everything output properly. It all revolved around handling the CTRL, SHIFT, and ALT keys, especially in nested states. I'm sure you are reading this wondering how hard could it be to record keystrokes and plop them into a string? Believe meâ"I originally thought it would be simple, too!
I started out by doing a finite state diagram of key actions so I could get a handle on the processing. I was originally going to include that state diagram for the keyboard handling, but the diagram is so complicated that it probably would have taken the entire MSDN Magazine art department three or four months to make it understandable enough to read. The other problem is that it would take two full pages to show. Yes, keyboard state is that confusing!
The first problem I encountered was what I will call the nested modifier states. As you know, the CTRL, ALT, and SHIFT keys are special modifier keys. CTRL+A, SHIFT+A, and ALT+A have three different meanings. If Windows allowed you to press only a single modifier at a time, my life would have been easy. In SendKeys format, a nested modifier looks like +(AB^(CD)). The + indicates a SHIFT, and the ^ indicates a CTRL. The following keystroke sequences produces that SendKeys string:
While there's some work involved with the previous example, try keeping the state on something like the following:
Try to manually walk through generating the SendKeys format for this and you will quickly run into the same issues I had to deal with. While you might think that I am going off the deep end attempting to handle strange cases, you would be surprised how many strange keystrokes you actually type. I whipped up a quick sample application that just recorded keystrokes; I constantly saw that I was typing keystroke sequences similar to the previous example mainly because I have naturally poor typing skills. While it would have been easy for me to simply say that you must type everything perfectly to record good keystrokes, I would have gotten tons of mail telling me that I copped out on solving the hard problems. Zen moments are cool when they are positive, but should be avoided if they are negative, especially when self-inflicted.
To get the nested modifier state processing to work took about four times as much testing as it took to develop. Up to that point, I was simply working with the A through Z characters.
For that previous key sequence example, the CRecordingEngine generates +(AB^(CD))^E. While it's not exactly the same key sequence in exactly the same order, that's the closest I could come with the SendKeys format.
After I got the A-Z characters with the SHIFT and CTRL modifiers working, I turned to handling the rest of the keyboard. If you have never had the joy of messing with virtual codes and scan codes, you don't know what you are in for. Additionally, I found that the keys the journal record hook received for some characters was quite a bit different than I expected.
The last time I messed around with keyboard processing at this level was way back in the old MS-DOSÂ® days. Consequently, I brought some misconceptions to the problem. For example, the first time I typed an exclamation point, I expected to see that exact character come through the journal record hook. What I got instead was a SHIFT followed by a 1. That's exactly what the keystroke sequence is on the US English keyboard. The problem is that I wanted any key sequences that I output to be mostly understandable. The SendKeys sequence +1 is technically correct, but you have to go through some mental gymnastics to realize that you are really looking at the ! character.
To make TESTREC.EXE as useful as possible, I needed to do some special processing so that the output strings were readable. Fortunately, these special case keys are limited to SHIFT modifiers. In SHIFTANDSPECIALKEYS.H I put them in an array, and in the CRecordingEngine::ProcessCtrlAltShiftUp processing I convert those values into the key values you expect.
The next problem I encountered was that certain characters are special keystrokes in the SendKeys format. Braces, brackets, and tildes in particular are special delimiters, so I needed to handle them. That special processing takes place when those keys are input and when I get a SHIFT up keystroke. There was nothing particularly difficult about it, but it complicated the processing.
After I thought I had most of the keystroke processing done, I spent nearly two full days recording and testing. After a bit of tweaking, I was finally handling every keystroke so that what I recorded was what I played back. The intense testing also turned up two small bugs in my PlayKeys routineâ"I was not properly restoring the modifier key states in two situations.
The final piece I tackled was the ALT+TAB processing. While I could have recorded ALT+TAB sequences and just used PlayKeys to do the work, I thought that would be a problem for your scripts because you probably do not want to have to restore the window z-order each time you run a script.
Near the top of RECORDINGENGINE.CPP is the CRecordingEngine::ProcessAltTabbing function. When you ALT+TAB to your heart's content, the CRecordingEngine state eats all the allowable keystrokes (ALT, TAB, and SHIFT) until you release the ALT key. Once you release the ALT key, I shift to the eDetermineFocus state and generate the appropriate code to set the focus when the next journal record hook notification of any kind comes through.
I record four different pieces of information about where the keyboard focus now sits because I want to force the focus right back to that specific location. The foreground window title is essentially the information in the title bar. The parent title is used to determine the instance I am supposed to look for. The Save As dialog appears at the top level. If multiple applications have the Save As dialog up, I need a way to determine which application to switch to. The child window title tells me the caption of the child window with focus. Finally, the child window ID helps me find windows that might not have captions. To get the child information, the GetGUIThreadInfo function makes figuring out the window with keyboard focus very easy. Since GetGUIThreadInfo only appeared in Windows NT 4.0 Service Pack 3, TESTREC.EXE will not run on earlier versions of Windows NT 4.0 or Windows 95.
After handling keystroke recording, updating TESTER.DLL with new methods was almost trivial. I added two new properties and a new method to the TWindow class:
The biggest change to TESTER.DLL was the addition of the SetSpecificFocus method to the TSystem class. This method is what the ALT+TAB processing will use to make sure that Tester sets the focus to the appropriate control. The only interesting part of the TSystem.SetSpecificFocus method, shown in Figure 3, is properly setting the focus on Windows 2000 and Windows 98. I have to call the AttachThreadInput function to synchronize the input threads for the two processes so I can call SetForegroundWindow and SetFocus. One thing I did learn is that if you call AttachThreadInput to attach, you must always call AttachThreadInput again to detach. If you do not detach, the attachee program will hang. As you look at Figure 3, you can see that I fully detach before raising any errors.
- Property ThreadId returns the thread ID for the window.
- Property Id returns the window ID.
- Method SetFocusTWindow calls SetFocus on the window.
One final note I want to make about TESTREC.EXE is that you must build with the January 2000 or later Platform SDK headers and libraries. After you install the Platform SDK, make sure that you change the directories in Visual C++ to use the new headers and libraries first. In the Directory tab of the Options dialog, select "Include files" in the "Show Directories for" combobox and move the directory where you installed the Platform SDK to the top. Do the same for the libraries by selecting "Library files" in the "Show Directories for" combobox.
What's Next for Tester? Overall, I am pretty happy with how TESTREC.EXE came out. As I have already alluded, there will have to be a Tester Three. While Tester Two is very useful as it stands, I need to add some functionality to make it complete and close to the perfect free regression testing utility.
The first thing I want to fix is the TESTREC.EXE output. Right now, TESTREC.EXE only produces VBScript output. I want to develop a simple plug-in interface so that you can have Tester scripts in any language.
The big feature to add is, obviously, mouse recording and playback. I have some ideas that might allow resolution-independent recording and playback. The one drawback I can see is that while I can get close to my goal, the playback on a different resolution might be off by a few pixels.
Another small change that should be easy enough to make is to have TESTREC.EXE and TESTER.DLL start handling the new keys for Windows 2000, such as VK_BROWSER_BACK.
The last feature I need to add is the handling of internationalized keyboards. Right now, TESTREC.EXE in particular is mostly hardcoded to the US English keyboard. Since many of you are from outside the United States, this limits the usefulness of Tester.
Wrap-up I hope that Tester Two will make your testing chores much easier. While the original Tester was useful, the lack of recording capabilities made building up a regression testing suite for your unit testing a pain. With Tester Two, you don't have any more excuses.
In my January 2000 column, I wrote a very useful utility, DbgChooser, that allowed you to pick the debugger to run when you encountered a crash. Karl Proese was kind enough to track down a bug in DbgChooser, which might have caused some problems.
The Windows NT and Windows 2000 Task Managers allow you to select a process to debug in the Processes tab. I had assumed that DbgChooser would always have a valid -e command-line option. The -e option indicates an event that a debugger must signal to show that it finished attaching to the process. When you choose to debug an application from Task Manager, the -e switch is used because the debuggee is not crashing. I would just pass the -e on to MSDEV.EXE, which would complain about the invalid command line and not debug the application. Included in this month's code distribution is an updated version of DbgChooser, which fixes the bug Karl found. I appreciate Karl and any readers who report bugs to me because I want to fix mistakes as soon as possible.
Da Tips! Celebrate the start of a brand new motorcycling season by sending your tips to me at firstname.lastname@example.org.
Tip 33 Korry Douglas writes: recently, my application was experiencing random deadlocks. Since I couldn't duplicate it at all, I added a new thread to the application that just waits on an event as soon as it starts. I gave a customer who could duplicate the deadlock a separate program called tickle that did nothing more than set the event that my special thread was waiting on. After tickle set the event, the special thread would wake up, suspend the main thread, dump the main thread's stack, and resume the main thread. Whenever the customer thought my application had hung, he would run tickle and I would get the magic call stack. Based on that I was able to figure out the problem.
Tip 34 Now that you have Tester to help do your regression testing, make sure you don't forget to check performance. When you build your testing scripts, you should record the start and stop times in order to get a baseline for how long various operations take. After your scripts run, take a look at the subsequent run's time versus your baseline; if you are more than 10 percent off, you need to stop and figure out why it slowed down. Performance problems creep into the applications a little bit at a time. If you start monitoring for them right up front, you can avoid many customer complaints in the end.
| John Robbins is an independent consultant and also teaches Windows debugging courses. He is the author of Debugging Applications (Microsoft Press, 2000). Reach John at http://www.jprobbins.com.|
From the June 2000 issue of MSDN Magazine.