Export (0) Print
Expand All
Add Support for Digital Ink to Your Windows Applications
Power to the Pen: The Pen is Mightier with GDI+ and the Tablet PC Real-Time Stylus
Expand Minimize

Ink and Agents on Tablet PC

 

Robert Crago
Revelation Computing Pty Limited

July 2005

Applies to:
   Microsoft Windows XP Tablet PC Edition Platform SDK 1.7
   Microsoft Agent 2.0

Summary: Learn how to use Microsoft Agent 2.0 in a Tablet PC application. Readers should be familiar with the Microsoft Windows XP Tablet PC Edition Platform SDK 1.7. (10 printed pages)

Click here to download the code sample for this article.


Contents

Introduction
Getting Started - Initialize Ink
Loading the Agent - Show Character
Measuring Shapes - Examine Shape
Measuring Straight Lines - Measure A Line
Drawing the Rulers - hRuler_Paint & vRuler_Paint
Measuring Horizontal Lines - Measure Width
Measuring Rectangles - Measure A Rectangle
Straightening the Line - Straighten Line
Biography

Introduction

If you're like me, when you think of Tablet PC you probably immediately think about such capabilities as pen input directly on the screen, handwriting recognition, landscape and portrait screen orientation, built-in wireless connectivity, speech recognition, and so on.

However there are a number of other freely available technologies that as developers, we might overlook. One of these technologies is the animated Microsoft Agent characters that we can sometimes use to enhance the user experience. The Microsoft Agent SDK hasn't been updated in a few years but it is still compelling technology when used appropriately.

Last year, I set out to develop an educational application for the Tablet PC that made use of its speech and handwriting recognition capabilities to conduct written conversations with young students. Though I originally intended to use Microsoft Speech API (SAPI) for the program's spoken output, I happened to read a copy of Clayton E. Crooks' book, Developing Tablet PC Applications. Chapters 19 and 20 alerted me to the possibility of coupling Microsoft Agent with handwritten input instead. Immediately I realized that this would present a much more compelling interface with which young students could conduct a conversation. (You can download this application in beta form from http://www.revComputing.com/TalkWrite.zip.)

The purpose of this article is to refresh your memory about the appropriate use of Microsoft Agent and to develop a simple application that shows how you can use agents and ink together on a Tablet PC to enhance the user experience. So let the fun begin...

Getting Started - Initialize Ink

In this article, I will discuss an application named Measure It that provides space for the user to draw a single stroke, representing either a horizontal line or a rectangle on the screen. Measure It incorporates a Microsoft Agent character that runs a tape measure along the line before proclaiming its length or around the rectangle to calculate and announce its area.

Click here for larger image.

Figure 1. Agent Peedy measuring a line.

I won't discuss every line of code (you can view the sample code for yourself) but just the areas where it deviates from standard Tablet PC applications.

I have used the excellent region feature of Microsoft Visual Studio .NET 2003 to divide the code into logical sections as seen in Figure 2. Throughout the article, I'll refer to the various regions as I explain the sample code.

Figure 2. Practical use of the region feature in Visual Studio .NET.

First, we must provide an area for drawing the line or rectangle using the pen. Visually, we lay out our form to include a WindowsForms panel control that covers most of the form to host the ink. We also include a label with written instructions above the panel and three radio buttons from which the user can select a measurement unit. In the code region Initialise Ink, we associate an InkOverlay object, named m_Ink, with our panel, panel1, when the form loads. Then we need to enable m_Ink so it can start collecting ink. Finally, we arrange for the main measurement logic of the application to commence only when the user lifts the pen after drawing something. The MouseUp event handler on the panel can do this. I could also have used CursorButtonUp event of m_Ink, but a conventional MouseUp event handler will do just as well.

private void Form1_Load(object sender, System.EventArgs e)
      {
         try
         {
            //Overlay ink on panel
            m_Ink = new InkOverlay(panel1);
            m_Ink.Enabled = true;

            //Start processing after a shape is drawn
            panel1.MouseUp += new MouseEventHandler(panel1_MouseUp);
         }
         catch (COMException)
         {
            //Thrown if InkOverlay can't be instantiated on target PC
            MessageBox.Show("Measure It can only run on a Tablet PC.");
            Application.Exit();
         }
      }

Listing 1. The Ink.MouseUP event handler.

Loading the Agent - Show Character

Next, we will add an agent character to the application and initialize it when required.

To use a program on a Tablet PC that supports agents, the following software must be installed:

Create a new Windows Forms application in Visual Studio .NET and open it at the forms designer. Now scroll down the Windows Forms section of the Toolbox and you should see the Microsoft Agent Control 2.0. If it's not checked, it's not present, so you can add it.

To add the Microsoft Agent Control 2.0 to your Toolbox

  1. From the Tools menu, click Add/Remove Toolbox Items.
  2. Click the COM Components tab.
  3. Select Microsoft Agent Control 2.0, and then click OK.

The Microsoft Agent Control is now listed in the Windows Forms section of the Toolbox.

Figure 3. The Microsoft Agent Control in the Visual Studio Toolbox.

Double-click the Microsoft Agent Control 2.0 in the Toolbox and the agent icon appears in the top left corner of your form. This control has no visual appearance on the form as such, but it does have properties. Select the agent's icon that appears on the form and view its properties. The Name property is set by default to AxAgent1. This property represents the name you must use to refer to the agent in your code. For Measure It, I changed the Name property to Agent.

Rather than initializing and showing the agent when the form loads, I chose to wait until the first shape is drawn. Otherwise the agent appears by default at the top left hand corner of the screen. If you examine the MouseUp event handler, you'll see the call on the ShowCharacter method, provided the click was inside the form's panel. A message appears if the click is outside the panel. Expand the Show Character code region and examine the code therein.

private Stroke ShowCharacter()
      {
         if (m_Character == null)
         {
            string agentPath = "c:\\windows\\msagent\\chars\\Peedy.acs";
            try
            {
               //Load Agent Character
               Agent.Characters.Load("Peedy", agentPath);
               m_Character = Agent.Characters["Peedy"];
               m_Character.LanguageID =0x409;            //English

               //Set to process any bookmark events in spoken output
               Agent.Bookmark += new AxAgentObjects._AgentEvents_BookmarkEventHandler(Agent_Bookmark);
            }
            catch (FileNotFoundException)
            {
               //No agent found
               MessageBox.Show("Please install " + agentPath + " first");
               Application.Exit();
            }
         }
         //
         if (m_Ink != null)
         {
            if (m_Ink.Ink != null)
            {
               if (m_Ink.Ink.Strokes != null)
               {
                  if (m_Ink.Ink.Strokes.Count > 0)
                  {
                     //Something has been drawn (only one stroke allowed)
                     Stroke lastStroke = m_Ink.Ink.Strokes[m_Ink.Ink.Strokes.Count - 1];
                     return lastStroke;
                  }
               }
            }
         }
         return null;
      }

Listing 2. The Show Character region.

Member m_Character is an instance of AgentObjects.IAgentCtlCharacterEx and in fact represents a physical on-screen manifestation of an agent. If m_Character is null, we haven't loaded our agent character yet and we need to do this once only.

What's the difference between the references to Agent and m_Character? Remember that "Agent" was the name we associated with the Agent 2.0 Control we dragged onto our form in the designer. This represents the agent server process that runs in a background thread and controls all agents. Once we have asked this server to load a specific instance of an agent using Agent.Characters.Load we need to be able to distinguish it from any other agents that are in use. This is where m_Character comes in. As you can see from the code, m_Character is really shorthand for a specific entry in the Agent.Characters collection. The collection index (in this case, "Peedy") is the same as the name parameter passed to Agent.Characters.Load(name, path) when the agent was loaded. Once the agent is loaded m_Character is the main object reference used when giving it instructions (such as Speak, MoveTo, and so on).

As well as loading the specific agent (and handling the exception thrown if it cannot be found) the Show Character region also creates a bookmark event handler—I'll give more details about this later.

A word about Microsoft Agent paths and filenames. You will notice that we have hard coded the path to Peedy's .acs file. This application uses a specific agent character as many of the animation movements are tied specifically to Peedy's size and wingspan. A more general application may allow the user to choose their favorite agent from either a File Open dialog box or a drop down list of .acs files available on the user's computer. For example, the TalkWrite Script Editor (not yet available for download from http://www.revComputing.com/TalkWrite) allows a teacher to choose the most appropriate agent for students to chat with, depending on the scripted conversation.

Measuring Shapes - Examine Shape

Once our MouseUp event handler has ensured that Peedy is loaded correctly, it is time to get on with it and measure the actual shape that the user has drawn. This takes place in the Examine Shape code region. At a high level, this code converts ink coordinates as used by the pen to screen coordinates as used by the agent and determines whether the shape drawn was a line or rectangle. Because the measurement animations are different for each of these shapes, either the MeasureALine() or MeasureARectangle() method is then called to animate the specific measurement process and announce the result. After either of these methods has returned, it just remains to tidy up and request another shape from the user.

Note   Shape recognition is not as easy as it looks. It is one of those operations that the human brain finds easy but is actually quite difficult to code. I thought the Windows XP Tablet PC Edition SDK properties PolylineCusps or BezierCusps would be an easy solution to this problem and return four cusps (that is, potential corners) for a rectangle and only two for a line.
However, a little experimentation quickly showed that a hand-drawn rectangle could return two, three, four, or more cusps depending on how carefully or carelessly the rectangle was drawn. More specifically, it depended on the angles of the corners. Corner angles close to 90 degrees were recognized as such, but the further they deviated from 90 degrees the less likely they were to be considered a corner. Even a hand-drawn straight line could be seen as having more than two cusps if it was too wiggly.
So I chose a simpler algorithm based on the distance between the start and end points of the shape. If these points are close together, the shape is considered to be closed (that is, a rectangle) otherwise it is a line. It's not perfect, but it will do. Any suggestions for an improved "shape recognition" algorithm on Tablet PC can be sent to me at robert@revComputing.com.

Measuring Straight Lines - Measure a Line

The task of measuring a straight line is handled by the MeasureALine method. MeasureALine optionally calls StraightenLine first to convert the user's shaky sloped line into a perfectly straight horizontal line. Then it calls MeasureWidth to run the tape measure along the line. MeasureARectangle calls MeasureWidth as well, along with its cousin MeasureHeight, to measure both sides of the rectangle. Once MeasureWidth has done its thing and announced the result, MeasureALine tidies up and returns.

In this application, a number of things are happening on different threads. Ink strokes are handled by one thread, the agent animation is handled by another thread, and the user interface events of the application as well as the code that follows are taking place on a third thread. If the thread dedicated to the agent's animation is processing Peedy's conversation, for example, and the user clicks Close, the application will start to quit but may leave the agent in an indeterminate state—often causing a COM exception. To prevent this, at Form1_FormClosing we set m_Quitting to true and check it before every significant agent operation to ensure we don't ask the agent to perform a task when it has already been terminated.

Drawing the Rulers - hRuler_Paint and vRuler_Paint

Our rulers are simply panel controls from the form designer that paint themselves with a border and minor and major tick marks. Every major tick mark is labeled. The horizontal and vertical controls, hRuler and vRuler, differ only in their orientation. All this work is done in their paint methods, hRuler_Paint and vRuler_Paint. See the Paint Rulers region for the code.

In these methods, the three radio buttons (InchesBtn, CMBtn, and PixelBtn) are interrogated to see what tick marks and label text to draw for each ruler. The panel's graphics properties, DpiX and DpiY, are interrogated to determine how many pixels represent an inch or centimeter.

Measuring Horizontal Lines - Measure Width

MeasureWidth is where the action begins. The purpose of this method is to display the beginnings of a ruler under the left hand side of the line and make it grow progressively longer as Peedy moves from left to right.

Animation is all about making small incremental changes sufficiently quickly in order to fool the eye into thinking the motion is continuous.

For each animation step, our application has simply to extend the width of the hRuler control by a small amount and refresh it. Then, hRuler_Paint does the rest. At the same time, Peedy is moved a little to the right every one tenth of a second via Thread.Sleep(100). To make the ruler appear from under Peedy's outstretched wing, a GestureRight animation is played before any movement starts and is of sufficient duration to last until the animation has ended.

Agent operations are asynchronous and queued. There is no guarantee that operations will start when requested or finish at any particular time. Further, no requested operation will start until the previous operation has finished. The only way to ensure your non-agent code doesn't start until after a particular agent operation has ended is to wait until that agent operation is complete. Fortunately, when long running agent operations are queued, they return a Request object which may be queried for the status of that request. This is where the utility method WaitForCompletion comes in. It continuously polls the Status property of the Request object until it indicates that the request has completed. To ensure other events keep happening while waiting, Application.DoEvents is called periodically. You'll find WaitForCompletion in the Utilities region of the code.

So MeasureWidth first creates a horizontal ruler that is positioned just under the left hand corner of the hand-drawn line. It is 48 pixels high but initially 0 pixels in width. As Peedy works in screen (global) coordinate space the stroke bounds rectangle (m_sb) has to be converted from panel-relative to screen-relative before the animation can begin. The For loop that follows performs the animation, advancing every tenth of a second.

         //Animate character along bottom of shape
         for (short x = (short)(screenB.Left); x < (short)(screenB.Right + 1); x+= 10)
         {
            if (m_Quitting)
            {
               return;
            }
            //Move agent a bit to the right
            m_Character.Left = x;

            //Extend width of ruler to the right
            hRuler.Width += Math.Min(10, (m_sb.Width - hRuler.Width) + 2);
            hRuler.Refresh();

            //Allow other threads to do their thing
            Application.DoEvents();
            Thread.Sleep(100);
         }

Listing 3. Animating the character along the length.

Coding realistic-looking animation is usually a tradeoff of either space or time. "How far to move over what time period?" is the question. I find that varying the distance and time interval by trial and error fairly rapidly produces a reasonable looking result. I'm my own worst critic when evaluating performance. Because I know what is going on under the hood, I tend to get caught up in all the fine detail rather than standing back and evaluating just the overall effect.

In each trip around the For loop, Peedy moves another increment to the right and that extra amount of the ruler is exposed. We cannot use the built in MoveTo agent animation to move Peedy because it is too slow. Instead we animate him ourselves by just making the character jump a little to the right each time as we change the Left property of m_Character. If the jump is small enough, the movement looks continuous. In doing animation this way, we can also perform other actions, such as extend the ruler and redraw it each time Peedy moves. The Math.min calculation ensures that the last movement is just to the end of the hand-drawn line when it is not a multiple of the minor tick mark length. If you make the correct tradeoff between space and time, Peedy's movement is smooth and life-like.

The final width of the ruler represents the width of the line in pixels. This width is converted to the real-world units indicated by the selected radio button. So Peedy now announces that width to the user (using selected units) and tidies up.

Measuring Rectangles - Measure a Rectangle

MeasureARectangle follows much the same pattern as MeasureALine except that there is a little more work to do. Once ExamineShape determines that the shape is a rectangle, MeasureARectangle measures and reports the horizontal edge first (MeasureWidth) and then the vertical edge (MeasureHeight). Finally Peedy flies to the center of the rectangle to announce the area before tidying up. Peedy writes for a moment, giving the impression of having to perform a handwritten calculation on paper to compute the rectangle's actual area.

MeasureARectangle twice makes use of the agent's bookmarks feature when Peedy is animated to speak and think. Bookmarks allow an action to be synchronized with a particular place in the agent's spoken (or thought) text. Otherwise it would only be possible to synchronize actions with the start or end of spoken text which is often not precise enough. Inserting a \Mrk=n\ tag in the string to be spoken will notify your code when that point has been reached through a Bookmark event. Remember that back in the Show Character code region, we set up an event handler for agent bookmarks? If you examine the Process Bookmarks code region, you will see that the code switches e.bookmarkID to handle each of the two types of spoken actions this application supports.

private void Agent_Bookmark(object sender, AxAgentObjects._AgentEvents_BookmarkEvent e)
      {
         Graphics g = panel1.CreateGraphics();
         switch (e.bookmarkID)
         {
            case (int)bookmarks.CROSS_HATCH:
               //
               g.FillRectangle(new HatchBrush(HatchStyle.DarkDownwardDiagonal, Color.LemonChiffon, Color.Blue), m_sb);
               break;
            case (int)bookmarks.STRAIGHTEN:
               StraightenRectangle();
               break;
         }
      }

Listing 4. Handling the two types of spoken actions.

The first of these bookmarks (Mrk=1) is used when pronouncing the replacement of the hand-drawn rectangle with its straightened counterpart. This ensures the rectangle changes shape just when the word "rectangle" appears on the screen in the phrase "I think I'll just straighten up this messy rectangle first." The second bookmark, Mrk=2, shades the rectangle just when the phrase "...this area..." is spoken at the conclusion of MeasureARectangle.

Straightening the Line - Straighten Line

An optional feature of Measure It is the ability to visually straighten the user's horizontal line before measuring it. After all, it is always the straightened line that is being measured, so it makes sense to make this obvious to the user. My idea here was to have Peedy jump up and down on the higher end of the line and then ride it down to the horizontal as it straightens. It's all done with a MoveTo agent animation (which produces the jumping effect if coming from underneath, followed by an animation loop that moves the straightened line gradually downwards towards the horizontal in five steps whilst simultaneously moving Peedy so that he appears to be sitting on, or just above, the line each time.

That's about it for Measure It. I hope you have been inspired to consider including agents, where appropriate, in your next Tablet PC application.

Biography

Robert Crago is an Australian developer based in Brisbane who was a finalist in the Microsoft Tablet PC Does Your Application Think in Ink? software development contest. He is particularly passionate about handwritten ink input and its application to teaching handwriting to beginning writers. Most of his development and consulting work is centered on pen-based hardware such as PDAs and Tablet PCs. Robert can be reached at robert@revComputing.com.

Show:
© 2014 Microsoft