Tablet 101 Column 2: Adding Context Awareness to Improve Handwriting Recognition

Frank Gocinski
Microsoft Corporation

July 2004

Applies to:
   Microsoft® Windows® XP Tablet PC Edition

Summary: Describes how context awareness can bias handwriting recognition and improve accuracy results. The article further describes the three current methods of adding context awareness to your application: the Context Tagging tool; the Factoid and WordList properties; and the SetInputScope API. Readers should be familiar with the Tablet PC Platform API. In addition, this article references the Tablet PC Platform SDK 1.7. (22 printed pages)

Download the sample code.


Using the Sample
Apply Some Context!
   Context Tagging Tool
      Let's Tag an Application
   Programming Context
      The Factoid and Wordlist Properties
      Set the Facts!
      Using a Word List


Well, this is my second Tablet 101 column, and I have to say it's been a lot of fun putting it together. The Tablet PC team appreciates your comments, and I encourage you to continue telling us what you want to see and how we are doing. This column is for you!

With this installment, we're going to take a closer look at how to add context awareness into your application. Context awareness enables you to prepare the recognizers for the probable results of the ink-to-text conversion process. You can associate an expected input format—such as a data type like phone number or date, a list of words, or a specific pattern of characters—with each data entry field in your application. Consider this scenario: the user is entering a date into a form and intends to handwrite the date into the control. By biasing the recognizers with context awareness, you can provide the pattern for valid dates that the recognizers use when performing handwriting recognition. In this way the recognizer is much more likely to return a valid date than an incorrect word.

Microsoft® Office System 2003 and Microsoft Internet Explorer are both applications in which context awareness is used. Let's experiment for a moment with Internet Explorer and see just how much the power of suggestion improves the recognition results.

In figure 1, you can see the new Web keys added to Tablet PC Input Panel for entering common URL strings such as http:// and www. I have tapped the key for http://, and Input Panel enters it into the Address bar.

Figure 1. Internet Explorer, Input Panel and common URL components

Figure 2 shows the features that indicate that Input Panel knows the Address bar is a URL field. These features are enabled because the Internet Explorer has assigned context to the Address bar. In figure 2, Input Panel

  • Recognizes the words I write as a URL.
  • Doesn't automatically place a space after a period.
  • Doesn't convert the forward slash into a letter L or a letter I.

This really makes surfing the Web on a Tablet PC much more pleasing than in previous versions.

Note   In Windows XP Tablet PC Edition 2005, the English, French, and German recognizers use context.

Figure 2. Input Panel recognizes the handwritten URL correctly

Using the Sample

I created the sample code that accompanies this column so that it runs in four different ways, corresponding to the three ways that you can apply context awareness programmatically—each of which I explain in this column—and one way in which no context awareness is applied. The method I've used for this is to employ #include statements. The #include statements are already in the source code, but they are commented out.

  • To run the sample application without context awareness, do nothing.
  • To run the sample application with InkEdit controls enabled, remove the comment marks from before the #define USEINKEDITCONTROL statement.
  • To run the sample application with word lists enabled, remove the comment marks from before the #define USEWORDLISTS statement.
  • To run the sample so that you can explore the use of the SetInputScope API, remove the comment marks from before the #define USESETINPUTSCOPE statement.

By running the sample code and turning on and off the different types of context awareness, you can explore the effects on recognition accuracy.

Apply Some Context!

You can apply context awareness in three different ways. I'll briefly introduce these mechanisms and then explain them further. For detailed information about how to select the best technique for applying context to your applications, refer to the Windows XP Tablet PC Edition Development Kit version 1.7.

  • The Context Tagging tool is a non-programmatic solution that supports applications that are not ink enabled. This tool ships in the Tablet PC Platform SDK 1.7. The Context Tagging tool enables you to identify the editable fields in the application, associating expected input with those fields. The recognizers use the association to increase recognition accuracy. You do not impact the binaries for the application and thus you do not need to redistribute them. Rather, the tool creates a context tagging file which is installed in the same directory as the application's executable.
  • The RecognizerContext object's Factoid and WordList properties are a programmatic solution for ink-enabled applications. If your application uses the InkEdit control then you can easily apply context by setting a value to one of 50 or so pre-defined hints, by associating a list of words with a field, or by supplying advanced pattern matching criteria in the form of a regular expression for a field. The binaries for the application are impacted and must be redistributed when programming against the RecognizerContext object.
  • The Win32 SetInputScope API can also be used for applications that are not ink enabled. The Win32 SetInputScope API is not exposed as a managed API. It is only offered as a Win32 API and suggests the hint directly to Input Panel, which in turn communicates the hint to the recognizers for you. Again, the binaries for the application are impacted and must be redistributed.
    Note   As of the release of Windows XP Tablet PC Edition 2005, the following recognizers support using the common input scopes with the SetInputScope API:
    • English (United States)
    • English (World Wide)
    • German
    • French
    Microsoft is evaluating whether support for the common input scopes for recognizers of East Asian characters is to be released in a future version of Windows XP Tablet PC Edition.

Context Tagging Tool

Let's talk details now. The Context Tagging tool ships in the Tablet PC Platform SDK version 1.7. It resides at c:\Program Files\Microsoft Tablet PC Platform SDK\Bin\Ctagger.exe. Using the Context Tagging tool, you can select a control and associate it with specific context information. The tool stores this association in a special context tagging file identified by the .ctm file extension. Input Panel uses the context tagging file to get the context information about controls in running applications. In turn, Input Panel informs the handwriting recognition engines of this context information, and the recognizer uses it to bias the handwriting recognition results it returns.

Context tagging files are specific to an application and have no effect on other programs.

Note   The context tagging file is an XML file. Do not manually edit the contents of a context tagging file. The schema of the context tagging file XML is subject to change and should not be created or modified outside of the Context Tagging tool.

Input Panel searches in the same directory as the application executable for a context tagging file and loads it, if present, when the application starts. Context tagging files conform to the Context Tagging File Schema and are validated on load by Input Panel. The only reliable way to create a valid context tagging file is by using the Context Tagging tool. Your context tagging file can be redistributed to Tablet PCs that are running your application.

You can associate hints, also known as input scopes, with a control by using the Context Tagging tool. An input scope is a defined set of words, numbers, punctuation, and syntactical orderings that are allowed within a given language. Applications can use input scopes to restrict the recognizer to a specific context. For example, an input scope of SMTPEmailAddress influences the recognition results by indicating that a specific field is an e-mail address. Thus, the field is likely to contain characters such as "@" and "_", and it may not contain characters such as "*" and spaces.

We'll try this process in a moment, but I first want to mention that you can find much more information about the Context Tagging tool, how input scopes are assigned, and, more importantly, what requirements your application must meet for the tool to be able to assign input scopes to your controls in the Help file associated with the Context Tagging tool in the Tablet PC Platform SDK 1.7.

Let's tag an application!

I've created a simple Windows forms application using Microsoft Visual Studio® .NET 2003 that manages my expenses when I am out on the road. This application has no context set for input fields (just yet, anyway). First, let's look at how the application operates without context awareness.

In Figure 3, I am about to enter an invalid date. You can see this in the preview of the recognition result in Input Panel. The recognizer has returned the best guess string for my poor handwriting. I certainly hope my code is up to the challenge of invalid data.

Figure 3. My application without context

Now, let's run the Context Tagging tool and select the appropriate input scopes for the fields. When you launch the Context Tagging tool, the dialog box shown in Figure 4 appears.

Figure 4. Welcome screen for the Context Tagging tool

You can create a context manifest for any executable file on your computer. Once you select an application and click New Context File, the selected application opens and the following dialog appears with the name of the application you are tagging in the dialog name.

Figure 5. The Context Tagging tool

Click the select control icon to open the Select Control dialog shown in Figure 6.

Figure 6. The Select Control dialog box

Click the Finder icon , drag it onto one of the controls in the application you want to tag, and then click OK. In the case of my expense report application, I dragged it onto the Date field.

Note   All controls in your application should be uniquely identified. This is a requirement for the Context Tagging tool, because the tool uses a combination of accessibility name, class name, and window identifier to uniquely identify the application. I set the accessibility name of all my fields in all my applications, which is a good practice for any developer to follow, ensuring the Microsoft Active Accessibility® architecture works properly for users that need to drive the application through alternative forms of input.

Once you have selected a control, go to step 2 in the main dialog box of the Context Tagging tool and choose an input scope for your selected control. The Context Tagging tool lists over 50 pre-defined input scopes that you can use and even allows you to create your own lists and unique patterns right in the tool.

Figure 7. Selecting an input scope

In figure 7, you can see that the selected input scope for my field, Date, is Date (full). To view the complete list of input scopes, see the online Help file available with the Context Tagging tool.

The next step is to define context awareness for the Expense Type field. For this field, the approved data is a known list of choices (such as lodging, travel, and so on). I can create a phrase list, which is a list of words that make up the possible results. Doing this in the Context Tagging tool is not only simple but quite powerful, too.

Figure 8. Building a Phrase List

Once you select the control to tag, select the Manage Phrase Lists tab in the main Context Tagging tool dialog box and create your phrase list. It's a simple process of creating a list of words to represent the appropriate choices. Either select an existing phrase list to edit or click Create New. To include a word in the phrase list, type the word in the Phrase field and then click Add. In the example in figure 8, I've created a phrase list named Expense Types that includes the words Travel, Meals, Conference, Lodging, and Other. It is similarly easy to replace, edit, or delete strings in a phrase list.

Figure 9. Choosing the phrase list

Figure 9 shows that the ExpenseTypes phrase list is now available to be associated with the ExpenseType control. Note that Coerce to Input Scope definition in step 2 is selected by default. This means that I will force Input Panel to return one of the members of the input scope as the recognized text, rather than merely using the strings in the input scope in addition to the system dictionary to search for the best recognized text.

After I tag all the relevant controls, I save the context file. The tool places the context tagging file in the directory where the application binaries exist. The Context Tagging tool names the context tagging file by appending an extension of .ctm onto the name of the executable being tagged. For example, if you create a context tagging file for MyExpenses.exe, the Context Tagging tool names the file MyExpenses.exe.ctm. This naming convention is important because it's what Input Panel looks for to load the .ctm when your application is running.

Figure 10. - The tagged application

Look at my application now. As you can see in figure 10, even when I write an obscure forward slash, because the context is set to the Date (full) input scope, Input Panel constrains the recognizers to known symbols for a date and resolves it properly to 6/04/04 –- pretty powerful!

Figure 11. Recognizing My Sloppy Handwriting

Remember my phrase list for the Expense Type field? I can write a word that doesn't really come close to the word Meals. Because Input Panel is using the phrase list to coerce to recognition results, it recognizes my sloppy writing correctly.

If your application has been released to market, cannot be modified, and meets the requirements of the tool, then you should use the Context Tagging tool to set context on the fields in the application. An application with a context manifest delivers an optimized experience for your Tablet PC customers. Even if you don't fully integrate your applications for ink and use of the pen, consider tagging your application and making the manifest files available for your customers on your Web site or as a downloadable update.

Programming Context

In addition to the Context Tagging tool, you can apply context programmatically. This is the preferred way to apply context, because it provides more control and granularity to you, the developer. Before we get started discussing details, let's step back and review the Tablet PC managed library object model so you have a basic understanding of some of the objects we'll use in the examples to follow.

Note   The managed library and Automation library object models are both very similar. This enables developers to transition easily between languages and opens Tablet PC integration for programmers of most imperative programming languages, including C, C++, C#, Microsoft Visual Basic® 6.0, Visual Basic .NET, J#, and even Java.

The managed library object model for the Tablet PC platform is fairly straightforward. The main components are represented in figure 12.

Figure 12. Tablet PC Platform API object model (simplified)

The goal of the Tablet PC Platform is really all about giving you programmable access to ink. The Tablet PC Platform enables the capturing of ink, manipulation of the ink once it's collected, and then interpretation of and rendering of the collected ink.

Let me give you a quick overview of some of the key parts of the Tablet PC managed library object model. The InkCollector and InkOverlay objects collect the ink from the digitizer and store it in an Ink object. The Ink object essentially acts as a container for a collection of Stroke objects. A Stroke object represents a single pen action sequence—that is, a pen down, pen move, and pen up sequence. The Stroke object exposes a DrawingAttributes property, which is a collection of attributes applied to the ink as it is being drawn. With these you can modify the ink's appearance, changing, for example, a stroke's width or color.

The Renderer object allows you to manage how ink is rendered to the display with methods including Draw, Measure, Rotate, Scale, and Move.

With the Tablet object, you can access information about the Tablet PC hardware and operating system. You can then enable and disable application features at run time based on this information.

You can discover which recognizers are installed on the computer through the Recognizer object. This object enables you to determine which languages are supported or what specific recognition functionality is at your disposal, such as object or text recognition. The Recognizer object also gives you access to the RecognizerContext object, which does the actual synchronous or asynchronous recognition work, returning the most probable result and list of possible alternates.

Now that we've laid the groundwork for the managed library, its time to survey in more detail the objects we'll use to apply context.

The Factoid and Wordlist Properties

The RecognizerContext object performs ink recognition, retrieves the most probable recognition result, and retrieves a list of alternate results. So, if there was ever an object to give some hints about recognition to, this is the one.

Factoids are content hints (just like input scopes) that can be given to the recognizer so that it may bias its results and achieve greater accuracy. Word lists are a type of factoid that contains a list of words which comprise the possible results of a particular recognition process. Both can be programmatically implemented to achieve some unparalleled recognition results.

The Tablet PC Platform SDK documents a collection of pre-defined factoids or input scopes. We assign the appropriate input scope or combination of input scopes to the Factoid property of the RecognizerContext object, which takes into consideration the suggestions when performing recognition.

Here's a list of known supported input scopes as of this writing.

  • URL (IS_URL)
  • File path (full) (IS_FILE_FULLFILEPATH)
  • File name (IS_FILE_FILENAME)
  • E-mail (user name) (IS_EMAIL_USERNAME)
  • Login name (IS_LOGINNAME)
  • Name (honorific) (IS_PERSONALNAME_PREFIX)
  • Name (first name) (IS_PERSONALNAME_GIVENNAME)
  • Name (middle name) (IS_PERSONALNAME_MIDDLENAME)
  • Name (surname) (IS_PERSONALNAME_SURNAME)
  • Name (suffix) (IS_PERSONALNAME_SUFFIX)
  • Postal address (full) (IS_ADDRESS_FULLPOSTALADDRESS)
  • Street address (IS_ADDRESS_STREET)
  • State or province (IS_ADDRESS_STATEORPROVINCE)
  • Country (abbreviation) (IS_ADDRESS_COUNTRYSHORTNAME)
  • Currency (amount only) (IS_CURRENCY_AMOUNT)
  • Date (full) (IS_DATE_FULLDATE)
  • Month (numerical) (IS_DATE_MONTH)
  • Day (numerical) (IS_DATE_DAY)
  • Year (numerical) (IS_DATE_YEAR)
  • Month (name) (IS_DATE_MONTHNAME)
  • Day (name) (IS_DATE_DAYNAME)
  • Number (positive digits only) (IS_DIGITS)
  • Number (full) (IS_NUMBER)
  • Character (IS_ONECHAR)
  • Telephone (country code) (IS_TELEPHONE_COUNTRYCODE)
  • Telephone (area code) (IS_TELEPHONE_AREACODE)
  • Telephone (local number) (IS_TELEPHONE_LOCALNUMBER)
  • Time (full) (IS_TIME_FULLTIME)
  • Time (hour) (IS_TIME_HOUR)
  • Time (minutes or seconds) (IS_TIME_MINORSEC)

These hints can be specified in one of two ways, as a suggestion which only influences the recognizers or as a forced mode which actually coerces the data recognition. We'll cover how to change the mode in the upcoming sample.

Set the Facts!

The Tablet PC Platform SDK exposes an ink-enabled text box called the InkEdit control. It is basically a TextBox control with an InkOverlay object attached to it. It looks and behaves like a text box and allows the user to write within the bounding rectangle. Remember from the first Tablet 101 article that it's the InkOverlay object that attaches to a window handle and puts up a transparent window which captures and renders ink for us fairly automatically. The collected ink can either be displayed and stored as ink or converted automatically to text. If your application uses InkEdit controls, you'll be able to apply context by setting the control's Factoid property. You can use a word list, too, although as you'll see it requires a bit more work to implement.

With the InkEdit control I have access to the Factoid property, and I can set it directly. As mentioned, the Factoid property is actually a property of the RecognitionContext object, and what InkEdit does is abstract all the programming logic you'd need to do the connection on your own – a true helper class.

// Programatically set the factoid for the date field.
ExpenseDate.Factoid = "(!IS_DATE_FULL)";
// Let's set the factoid for the expense amount to currency.
ExpenseAmount.Factoid = "(!IS_CURRENCY_AMOUNTANDSYMBOL)";

While we're at it, I'll also tell the description field to display ink as ink -- another cool feature of the InkEdit control.

ExpenseDescription.InkInsertMode = InkInsertMode.InsertAsInk;

And finally, I'll set the drawing attributes to ignore the pen pressure because the resultant image is a bit cleaner when I keep ink as ink.

DrawingAttributes da = new DrawingAttributes();
da.IgnorePressure = true;
ExpenseDescription.DrawingAttributes = da;

Let's run the application now with its new ink-enabled user interface.

Figure 13. The InkEdit control converts handwriting to text

You can see the behavior of the InkEdit control: it allows me to write some ink right in the control, and then it converts the ink to text after a variable timeout has expired. The description field displays the ink as ink while programmatically setting the factoid value.

Using a Word List

Factoids do a great job of improving recognition for common data types. Regular expression support covers most of the remaining problem space except for application-specific types of content, and for this there are word lists. The WordList property on the RecognizerContext object is a list of strings and instructs the recognizer to use this predefined set of strings as words that are likely to appear in the recognized text. Like a factoid, the WordList property can be either suggested or forced by setting the RecognitionFlags property on the RecognizerContext object. Word lists are the programmatic equivalent to the phrase list we defined in the Context Tagging tool.

Word lists create what is called an application dictionary. To create an application dictionary, it is necessary to set the WordList property of the RecognizerContext object. The InkEdit control does not expose the RecognizerContext object, so it is not possible to directly set the WordList property of the InkEdit control's RecognizerContext object.

An application dictionary is just that: its words are only used for a specific application. They are not available to every application on a Tablet PC. The system dictionary contains words available to all applications running on the Tablet PC, and adding words to that dictionary also aids in improving recognition. Microsoft has a Dictionary Tool for Tablet PC available for download. In addition, there are 3rd-party precompiled dictionaries that you can use for your organization, such as the ones available from abletFactory. Still others are listed in the Tablet PC Applications Catalog.

Let's modify my application to use a word list for the Expense Type field. I want to use the words "Travel," "Meals," "Conferences," "Lodging," and "Other." The following code sample will demonstrate the steps we need to take.

First, let's create the word list.

#if USEINKEDITCONTROL    // Use InkEdit controls. 
#if USEWORDLISTS   // Constrain the Expense Type recognition 
to a known list of results.

private void Form1_Load(object sender, System.EventArgs e)

//Create a WordList that contains the application dictionary.
wl = new WordList();

The next step is to add an event handler to respond to the Recognition event of the InkEdit control.

ExpenseType.Recognition += new

Now, let's create a RecognizerContext object to perform the recognition. I attach the word list to it and set the RecognitionFlags property to coerce the recognizer to use only the words in the list.

// Create a RecognizerContext object to set up our defaults and
 attach the word list.
   rc = ExpenseType.Recognizer.CreateRecognizerContext();
   rc.Factoid = Factoid.WordList;
   rc.RecognitionFlags = RecognitionModes.Coerce;

//Set the RecognizerContext WordList to the newly created WordList.
   rc.WordList = wl;

Next, we create an event handler to respond to the RecognitionWithAlternates event. The event handler is called whenever the RecognizerContext object is called to perform recognition.

//Create an event handler when this RecognizerContext object is called
 to recognize ink.
   rc.RecognitionWithAlternates += new 

Because Wordlist is not a property of the InkEdit control, to use it you must respond to the recognition events, take the ink out of the InkEdit control and pass it to the RecognizerContext object, store the results, and then pass the results back to the InkEdit control. I know it sounds complicated, but it's not that bad, and it gives you good exposure to how the recognition events fire.

The following code sample demonstrates the event handler that responds when the InkEdit control, ExpenseType, receives a request to recognize the ink it has collected.

private void ExpenseType_Recognition(object sender, 
Microsoft.Ink.InkEditRecognitionEventArgs e)
   // When the InkEdit contol for ExpenseType begins the process 
   // of recognition, this event will fire. 

// We store the start of the selection in wlSelStart.
   wlSelStart = ExpenseType.SelectionStart;
// We store the length of the selection in wlSelLen.
   wlSelLen = e.RecognitionResult.TopString.Length;

   // Copy the strokes from the InkEdit control into the new
   // RecognizerContext object.
   rc.Strokes = e.RecognitionResult.Strokes.Ink.Clone().Strokes;
// Call the background recognizer. 

Once the background recognition completes, it raises a completion event and the next event handler responds, entering the recognized text in the Expense Type field.

private void rc_Recognition(object sender, 
Microsoft.Ink.RecognizerContextRecognitionWithAlternatesEventArgs e)
   // rc_Recognition is called once the background recognition is complete.
   // Now we need to set the insertion point in the InkEdit control.
   ExpenseType.SelectionStart = wlSelStart;
   ExpenseType.SelectionLength = wlSelLen;
   // Insert the result from the new RecognizerContext object 
   // into the InkEdit control.
   ExpenseType.SelectedText = e.Result.TopString; // The most probable answer.

   // Reset the selection pointer after the input. 
   ExpenseType.SelectionStart = ExpenseType.SelectionStart +    

When I run the application and write anything even close to one of the words in the word list, I'll typically get a match. Take note of figures 14 and 15.

Figure 14. Enter anything remotely close to the word "Travel"

Figure 15. The word list helps the recognizers make the match

As a complement to this article, Arin Goldberg created a dynamic context tool that will shortly be available for download from the Tablet PC developer center. Arin's sample shows how you can dynamically use context awareness to affect the operation of your program.


The SetInputScope API is available starting with the Tablet PC Platform SDK 1.7 and is the final programmatic approach to setting context awareness that I want to talk about in this column. SetInputScope associates a window with some combination of one input scope, multiple input scopes, or one phrase list. This method is what you will use if your application is either a managed application that is not ink enabled, or a native application built with C, C++, or Visual Basic 6.0 and that does not use the InkEdit controls.

To best support your Tablet PC customers with context awareness in applications that are not ink enabled, look at adding programmatic support for relaying the context information directly to the Input Panel from within your application. That's where SetInputScope is used. To call SetInputScope, you need to first define the input scope then make a formal Win32 call to the SetInputScope function.

If you are not familiar with calling Win32 APIs from a managed environment, I suggest you check out Eric Gunnerson's article on MSDN titled Using Win32 and Other Libraries or see the excellent white paper written by John Robbins of Wintellect that discusses SetInputScope in much more detail. John has even provided a nice abstraction that will really simplify the process. His paper will be available shortly on the Tablet PC developer center.

After removing the comments from the directive, #define USESETINPUTSCOPE, to build the application using standard text boxes that will receive input from Input Panel, I define the DllImport to the Msctf.dll which is where SetInputScope lives.

   [DllImport ( "Msctf.dll", CharSet = CharSet.Unicode  ) ]
   private static extern Int32 SetInputScope ( IntPtr hWnd, int 
inputscope  ) ;

With this defined, I can now make a call directly to SetInputScope in my code.

   //   IS_DATE_FULLDATE                  = 22 ,
   Int32 ret = SetInputScope ( ExpenseDate.Handle , 22 ) ;

That's all there is to it. At form load, I can establish the run-time relationship between my TextBox control and its input scope. Input Panel stores this information in a run-time data storage mechanism called the Text Services Framework, and every time the text box is loaded, so is the Input Scope.

The following C++ snippets illustrate how to set an input scope, where HWND is the window of the input field, typically a TextBox control. The parameters for the call are fully explained in the Tablet PC Platform SDK 1.7.

   InputScope scope = IS_EMAIL_USERNAME;
   SetInputScopes(hwnd, &scope, 1, NULL, 0, NULL, NULL);

You can limit the scope of the recognizer's vocabulary to a list of words or phrases by using a phrase list. This causes the recognizer to coerce to the input scope.

SetInputScopes(hwnd, NULL, 0, { "blue", "green" }, 4, NULL, NULL);

For more information about the SetInputScope API, see the Tablet PC Platform SDK 1.7, currently in beta.


I hope you are ready to try applying some context awareness in your applications, now that we've discussed how to use the Context Tagging tool, the Factoid and WordList properties, and the Win32 SetInputScope API to do so. Your end users will appreciate it, and you'll find that the user experience will be much improved if you can take steps to bias recognition and improve the recognition results within your applications.

As always, remember to