Using Microsoft Tablet PC Input Panel Correction in Custom Recognition Scenarios


Stefan Wick
Microsoft Corporation

February 2005

Applies to:
   Microsoft Windows XP Tablet PC Edition 2005
   Microsoft Tablet PC Platform SDK

Summary: This article describes how to use the Microsoft Windows XP Tablet PC Input Panel's text correction capabilities in applications that collect and recognize ink outside of the Input Panel. The corresponding samples are written in C#. (11 printed pages)

Click here to download the code sample for this article.


Using the Samples


The Microsoft Windows XP Tablet PC Input Panel provides a great way to insert and correct handwritten text in Windows applications. Applications that are targeted specifically for usage on a Tablet PC, however, often strive to provide a more paper-like user experience that allows writing directly onto the application without opening an input utility. The Tablet PC Platform SDK 1.7 enables the creation of such applications that collect and recognize Ink on their own surface, but it doesn't provide a built-in correction mechanism for collected handwriting. The combination of a custom inking surface and the Input Panel's correction interface can increase the usability of an application and make handwriting input more efficient and accurate.


The Tablet PC Input Panel uses the Windows Text Services Framework when inserting handwritten text into an application, and stores the Ink together with the text in the target control. This allows for post-insertion correction when the user comes back and wants to make changes to the already inserted text.

The Tablet PC Platform SDK 1.7 provides an easy way to do the same with Ink collected in your application on your custom inking surface: the InkEdit control. This control provides the means to store text and the associated Ink in a way that is compatible with the Input Panel, so the user can use it to perform corrections on inserted handwriting. It is important to note that you don't actually have to use InkEdit to collect the Ink. You can disable Ink collection in the InkEdit control altogether (which essentially makes it a RichTextBox control) and programmatically insert Ink objects that have been collected elsewhere (using InkOverlay, RealTimeStylus, or other mechanisms, for example). The SelInks property provides the means to insert an array of Ink objects into the text stream of InkEdit's underlying RichEdit control. Inserting an Ink object into an InkEdit control using the SelInks property will trigger the following operations:

  • The handwriting will be recognized according to the control's Recognizer and Factoid properties.
  • The result will be displayed respecting the control's InkInsertMode property.
  • The Strokes will be associated with the result's text range in a way that is compatible with the Input Panel.

Now when the user brings up Input Panel on the inserted text range, the full correction experience, including recognition alternates, will be available.

Using the Samples

In order to compile the sample source code, you must have Microsoft Visual Studio .NET 2003 and Microsoft Tablet PC SDK 1.7 installed on your computer running Microsoft Windows XP Tablet PC Edition 2005. Note that the samples will not work on other operating systems.

The solution contains four projects:

  1. Sample1—A Windows application that demonstrates the basic concept of how to use the Tablet PC Input Panel's text correction capabilities for Ink collected outside of Input Panel.
  2. Sample2—A simple form-filling application that collects Ink on a paper-like User Interface (UI) and uses the Input Panel for correction purposes.
  3. Sample3—A note-taking application that allows taking notes like on paper and uses the Input Panel to correct the notes before converting them to text.
  4. SampleControls—A collection of controls used by Sample2 and Sample3.

In order to run these sample applications within Visual Studio, make sure you set the desired project as "StartUp Project," by selecting it in the Solution Explorer and then selecting "Set as StartUp Project" from the "Project" menu.

Sample 1—The Basic Concept

The basic concept described in this article is implemented in the Click handler of the "Recognize" button of Sample1. The code applies the user supplied input scope to the Factoid property and then assigns the collected ink to the SelInks property:

private void btnRecognize_Click(object sender, System.EventArgs e)
   // apply the input scope to the InkEdit control
   // you can use the members of the Factoid class,
   // a supported regular expression or one of the
   // new predefined input scopes (e.g. "(!IS_URL)")
      theInkEdit.Factoid = tbInputScope.Text;
      // fall back to "DEFAULT" if an unsupported
      // Input Scope has been supplied
      theInkEdit.Factoid = Factoid.Default;
      tbInputScope.Text = theInkEdit.Factoid;

   // assign the ink from InkPicture to InkEdit
   // which will perform the recognition automatically
   theInkEdit.Text = "";
   theInkEdit.SelInks = new Ink[]{theInkPicture.Ink};

Once the collected Ink has been inserted, the user can set the insertion point with the Stylus on the recognition result and bring up the Input Panel in order to correct the text if necessary:

Figure 1. Using the Tablet PC Input Panel for text correction

Sample 2—Form Filling

In the second sample application, the previous concept is being applied in a more realistic scenario that allows writing on a bank check like on a piece of paper. After a timeout, the collected Ink is inserted as text into text entry fields (such as the Date field) or left as Ink for Ink entry fields (such as the signature field).

The bank check UI has been built with controls from the SampleControls project:

  • The writing area on top of the check is a FreeFormPanel, which derives from the Panel control and has an InkOverlay attached.
  • The text entry fields are RichEditBox controls that derive from InkEdit.
  • The signature field is an InkLabel control that derives from the standard Label control.

When the user starts filling out the form, a two second timer is started. The timer is being reset whenever new Strokes are added. Once it expires, the Ink is divided up into segments using the Divider object. The Strokes for each segment are then associated with the nearest control. If it is a RichEditBox, the Strokes will be inserted into the text control, if it's an InkLabel, the Strokes will be rendered permanently onto the panel.

private void theTimer_Tick(object sender, System.EventArgs e)
   // this event fires when the user has not drawn any new
   // strokes in the past two seconds; it then processes the 
   // newly collected ink

   // if user is still writing restart the timer
   if (theOverlay.CollectingInk)

   // create a copy of the ink and delete the strokes in the InkOverlay
   // so more ink can be collected while recognition is pending
   Ink inkCopy = theOverlay.Ink.Clone();

   // Divide the ink into its segments/words
   Divider div = new Divider(inkCopy.Strokes);
   DivisionUnits units = div.Divide().ResultByType(InkDivisionType.Segment);

   // Insert each segment into the nearest respective edit box
   foreach (DivisionUnit unit in units)
      Control ctrl = NearestControl(unit.Strokes);
      if (ctrl is RichEditBox)
         // associate the edit box with the current unit's strokes
         // collection; this will also convert the strokes to text
         // in the control
         ((RichEditBox)ctrl).Strokes = unit.Strokes;
      if (ctrl is InkLabel)
         // add the strokes to the ink-as-ink layer so they will be
         // displayed permanently
         DrawingAttributes da = theOverlay.DefaultDrawingAttributes.Clone();
         da.Color = Color.Blue;
         inkAsInk.AddStrokesAtRectangle(unit.Strokes, unit.Strokes.GetBoundingBox());

The following screenshots illustrate the UI before and after the timer has expired:

Figure 2. The User Interface before the timer has expired

Figure 3. The User Interface after the timer has expired

The user can now choose to switch into correction mode, which will disable the inking surface on the check and enable the Input Panel for each text control. This will allow the user to change the capitalization of "kitchen," or to make other corrections:

Figure 4. Input Panel correction mode

Sample 3—Note Taking

The third sample application shows how to use the Input Panel correction capabilities in a note-taking application. In this type of application the user typically leaves the handwritten notes as Ink. However, for interoperability with other applications, the notes sometimes have to be converted to text. The sample shows how to let the user view and correct the recognition results for handwritten notes by selecting the respective Strokes.

The Ink collection surface here is a plain InkPicture control. The application also uses hidden RichEditBox controls from the SampleControls project in order to store the recognized text behind the scenes. Whenever the selection in InkPicture changes, we'll look for a RichEditBox that is associated with the selected Strokes collection. Then we set focus to the hidden control and allow the Input Panel to do its correction job.

private void theInkPicture_SelectionChanged(object sender, System.EventArgs e)
   // when the selection changes in InkPicture, we need to find
   // the edit box that is associated with this selection
   selectedEdit = null;
   if (theInkPicture.Selection.Count == 0)
      // if no strokes are selected it means that the user has
      // been dismissed the selection so we just need to refresh
      // the document

   foreach (RichEditBox edit in editControls)
      // find the edit box that is associated with the
      // selected set of strokes
      if (StrokesEquals(edit.Strokes, theInkPicture.Selection))
         // if the strokes in the edit box are equal to the
         // selected strokes we have found the right edit box
         selectedEdit = edit;

   if (selectedEdit == null)
      // no edit box has been found that matches the selection;
      // we get here in case of sub-word or multi-word selection;
      // in this case we just dismiss the selection
      theInkPicture.Selection = theInkPicture.Ink.CreateStrokes();

   // set focus to the edit box and select its content to enable
   // the TIP correction experience
   selectedEdit.Select(0, 0);

The hidden edit boxes are updated with the most recent recognition results whenever the user switches into Preview or Correction mode. It's important to not override any correction that has been done previously by the user—unless the Strokes have actually changed.

private void UpdateRecoWords()
   // this function updates the array of edit boxes to
   // ensure that there is one edit box per word in
   // the document; each edit box initially contains
   // the recognition result (incl. alternates); it
   // also stores the modified text in case the user
   // has corrected the result via TIP

   // display hourglass cursor while recognizing the ink
   Cursor = System.Windows.Forms.Cursors.WaitCursor;

   // use InkDivider to get the strokes collection
   // for each segemnt in the document
   Divider inkDivider = new Divider();
   inkDivider.Strokes = theInkPicture.Ink.Strokes;
   DivisionUnits inkUnits = inkDivider.Divide().ResultByType(InkDivisionType.Segment);
   RichEditBox []newEditControls = new RichEditBox[inkUnits.Count];
   Graphics g = theInkPicture.CreateGraphics();
   for (int i=0; i<inkUnits.Count; i++)
      // loop over all segments and associate the strokes
      // with an edit box unless they are already
      // associated with an edit box
      Strokes strokes = inkUnits[i].Strokes;
      if ( (newEditControls[i] = FindEdit(strokes)) == null)
         // if there is no edit box for this segment
         // create a new one
         newEditControls[i] = new RichEditBox();

         // set the control's location according to the
         // location of the strokes
         Rectangle rect = strokes.GetBoundingBox();
         Point pt = new Point(rect.X+rect.Width/2, rect.Y+rect.Height);
         theInkPicture.Renderer.InkSpaceToPixel(g, ref pt);
         newEditControls[i].Location = pt;

         // insert the strokes from this segment
         newEditControls[i].Strokes = strokes;

         // make control invisible
         newEditControls[i].Size = Size.Empty;

         // make it a child of the InkPicture
         newEditControls[i].Parent = theInkPicture;

         // listen to any text changes when the user
         // performs correction
         newEditControls[i].TextChanged += new EventHandler(Sample3_TextChanged);

         // add it to the list of controls that need
         // to be disposed when we're done
   // replace current collection of edit boxes with the
   // new collection of edit boxed
   editControls = newEditControls;
   // set cursor back to default to indicate we are done
   // recognizing
   Cursor = System.Windows.Forms.Cursors.Default;

   // clean up

The following screenshots illustrate the sample application in both inking and correction mode.

Figure 5. Sample Application in Inking mode

Figure 6. Sample Application in Correction mode


  • An Ink object can be inserted programmatically into an InkEdit control using the SelInks property.
  • The Strokes will be recognized automatically according to the current Recognizer and Factoid properties.
  • The best recognition result will be displayed as text, and alternates will be accessible through the Input Panel correction UI.
  • The combination of the preceding features can be used to combine custom Ink collection scenarios with standard Input Panel correction experiences.