Creating an Ink-Enabled ListBox Component for Tablet PC

  • Article

 

Leszynski Group, Inc.

March 2006

Applies to:
   Microsoft Tablet PC Platform SDK
   Microsoft Windows XP Tablet PC Edition 2005
   Microsoft Visual Studio .NET 2003

Summary: This article and sample code demonstrate how to write your own custom component that extends the existing ListBox control to support ink collection and recognition. (17 printed pages)

Click here to download the code sample for this article.

Contents

Introduction
InkListBox Overview
Building InkListBox
Ink Recognition
Modifying List Box Items
Extending InkListBox
Conclusion
Biography

Introduction

This article and sample code provide an example of how to write a custom control for use in Windows Forms applications intended for use on Tablet PC. The sample code creates a component called InkListBox that extends the existing ListBox control to make it ink enabled. The sample demonstrates how to:

  • Extend the functionality of an existing control.
  • Perform ink recognition.
  • Coerce recognition of ink strokes by using word lists and factoids.
  • Expose a control's custom properties in a property grid.
  • Provide a simple and elegant user experience for new Tablet PC users.

The custom control is contained in an assembly named Microsoft.Samples.TabletPC.InkListBox.dll that you can use in your applications as-is or with customization.

The sample was created in Visual Studio 2003 (.NET Framework 1.1) using C#. The sample also includes compiled binaries that can be demonstrated without Visual Studio. For installation instructions and demonstration uses, see Using the InkListBox Sample in the root folder of the InkListBox sample.

The sample code can be converted successfully to Visual Studio 2005 without errors but has not been extensively tested with this version.

InkListBox Overview

The goal of the InkListBox sample is to provide a simple, yet realistic, example of the type of user interface controls you can build with the Tablet PC SDK to improve and customize the user experience in software solutions. The sample demonstrates that with only a modest amount of effort you can combine controls from the .NET Framework and the Tablet PC platform to create a customized data entry experience.

InkListBox provides a data entry experience where the Tablet PC pen provides an alternative to the keyboard for the selection of one or more items from a list of data. InkListBox gives you control, through object properties or source code, of the following elements of the user experience:

  • Setting ink collection on or off.
  • Enabling or disabling gestures for list navigation.
  • Performing recognition on single or multiple words.
  • Setting the recognition timeout.

Figure 1 shows the InkListBox component on a Windows Form object. An ink collector overlays the list box and provides support for the Tablet PC pen. The user writes directly on the list, and the control selects the first matching list item.

Figure 1. InkListBox component on a form.

Building InkListBox

InkListBox derives from the standard ListBox control in order to:

  • Provide the simplest developer experience in the Visual Studio designer.
  • Build on existing properties and events for list management.
  • Enable developers to use InkListBox as a drop-in replacement for the ListBox controlin existing applications that are being upgraded to support mobility.

The following code shows the declaration for the InkListBoxControl and the constants that manage default values for its custom properties:

public class InkListBoxControl : System.Windows.Forms.ListBox
{
   #region Constants
   
   // Constants.  These values will be set when an InkListBox is added to 
   // a form and can then be changed either by the Property Grid in design 
   // view, or programmatically when an application is running.
   private const int RecognitionTimeoutDefault = 1500;
   private const bool IgnoreTapGesturesDefault = false;
   private const bool PageUpDownGesturesEnabledDefault = false;
   private const bool SingleWordModeDefault = true;
   private const bool InkEnabledDefault = true;

   #endregion

InkListBoxControl declares two private members related to ink collection and recognition:

  • InkCollector – Supports the collection of ink strokes on the surface of the underlying ListBox control.
  • RecognizerContext – Provides support for gestures and recognition of words that correspond to the collection of items on the list.

InkListBoxControl does not perform most of its initialization in its constructor because the control cannot determine at that time if it is in design mode. Instead, the control's initialization takes place the first time that the HandleCreated event is raised. At that time the control ensures that it is not in design mode and that handwriting recognizers are installed. If these criteria are not met, the initialization is skipped and the control behaves as the standard ListBox. On a Tablet PC, the test for handwriting recognizers always succeeds, but the test fails if the Tablet PC SDK is installed on a non-Tablet PC computer but the Microsoft Windows XP Tablet PC Edition 2005 Recognizer Pack is missing.

/// <summary>
/// This member overrides ListBox.OnHandleCreated.
/// </summary>
/// <param name="e">An EventArgs that contains the event data.</param>
/// <remarks>
/// This handler is overridden to perform initialization tasks. These tasks 
/// are performed here instead of in the constructor since the DesignMode 
/// state isn't available when the constructor executes.
/// </remarks>
protected override void OnHandleCreated(EventArgs e)
{
   if (!initializationDone)
   {
      // Determine whether any recognizers are installed. If there are
      // no recognizers, the control reverts to working as a normal 
      // ListBox.
      recognizersInstalled = false;
      try
      {
         Microsoft.Ink.Recognizers recognizers = new Recognizers();
         recognizersInstalled  = (recognizers.Count > 0);
      }
      catch
      {
         // Instantiation of the Recognizers class may throw an exception 
         // if no recognizers are installed, so we can ignore this 
         // exception and leave recognizersInstalled in its default state 
         // of false.
      }

      if (recognizersInstalled && !DesignMode)
      {
         // Set up InkCollector.
         inkCollector = new InkCollector(this);
         inkCollector.Enabled = InkEnabledDefault;
         inkCollector.CollectionMode = CollectionMode.InkAndGesture;
         inkCollector.Gesture += new InkCollectorGestureEventHandler(inkCollector_Gesture);
         inkCollector.MouseDown += new InkCollectorMouseDownEventHandler(inkCollector_MouseDown);
         inkCollector.Stroke += new InkCollectorStrokeEventHandler(inkCollector_Stroke);

         // Set up recognition.
         // The RecognizerContext constructor returns the default recognizer context.
         recognizerContext = new RecognizerContext();
         recognizerContext.Factoid = Microsoft.Ink.Factoid.WordList;
         recognizerContext.WordList = new WordList();
         recognizerContext.Strokes = inkCollector.Ink.CreateStrokes();
         recognizerContext.Recognition += new RecognizerContextRecognitionEventHandler(recognizerContext_Recognition);

         // Apply default values for properties.
         timer.Interval = RecognitionTimeoutDefault;
         SetInkEnabled(InkEnabledDefault);
         SetIgnoreTapGestures(IgnoreTapGesturesDefault);
         SetPageUpDownGesturesEnabled(PageUpDownGesturesEnabledDefault);
         SetSingleWordMode(SingleWordModeDefault);
      }

      initializationDone = true;
   }

   base.OnHandleCreated (e);
}

InkListBox exposes its custom properties to the Properties Window in Visual Studio by including attributes for each property. For example, the following code shows the attributes for the InkEnabled property:

[Category("Ink")]
[Description("Enables or disables ink collection.")]
[DefaultValue(InkEnabledDefault)]
public bool InkEnabled

Figure 2 shows how these attributes provide a category and description in the Properties Window.

Figure 2. InkListBox.InkEnabled in the Properties Window

Ink Recognition

Ink recognition is enabled in InkListBox when all of the following conditions are met:

  • Recognizers are available.
  • The InkEnabled property is set to true.
  • The SelectionMode is not set to MultiExtended.

InkListBox performs both handwriting recognition and gesture recognition. In OnHandleCreated, the ink collector's CollectionMode is initially set to InkAndGesture, which enables both Stroke events and Gesture events. Gesture events are handled by the inkCollector_Gesture method, which performs the appropriate action based on the gesture that was recognized:

  • Gesture down – Pages down on the list (sends the PGDN key).
  • Gesture up - Pages up on the list (sends the PGUP key).
  • Gesture tap – Selects a list item.
  • Gesture double-tap – Selects a list item.

The following code sample shows how the gestures are implemented in the InkListbox control:

//
// inkCollector_Gesture
//
// Occurs when an application gesture is recognized.
//
// Parameters:
//  sender - The source InkCollector object for this event.
//  e - The InkCollectorStrokeEventArgs object that contains the event data.
//
private void inkCollector_Gesture(object sender, InkCollectorGestureEventArgs e)
{
    // This event can be fired on a background (non-UI) thread, 
    // which can cause unpredictable behavior when accessing UI objects. 
    // Use the Invoke method if necessary to handle the event on the UI thread.
    if (InvokeRequired)
    {
        Invoke(new InkCollectorGestureEventHandler(inkCollector_Gesture),
            new object[] { sender, e });
        return;
    }

    // Prevent exceptions from propagating back to the sender -- the sender could ignore them, 
    // making it much harder to debug associated problems, and other event subscribers could 
    // be prevented from being called, leaving the system in an unanticipated state.
    try
    {
        if (e.Gestures[0].Confidence == RecognitionConfidence.Strong)
       {
          switch (e.Gestures[0].Id)
          {
            case ApplicationGesture.Down:
               InvalidateStrokesRect(e.Strokes);
               SendKeys.Send("{PGDN}");
               break;
            case ApplicationGesture.DoubleTap:
            case ApplicationGesture.Tap:
               Point inkPoint = e.Strokes[0].GetPoint(0);
               using (Graphics g = CreateGraphics())
               {
                  // Find the item corresponding to the tap.
                  int index = IndexFromPoint(Convert.InkSpaceToPixel(g, inkPoint));

                  // Simulate the functionality of the current SelectionMode.
                  // Note that we don't need to consider the case of MultiExtended 
                  // mode since ink collection is disabled in that mode.
                  if (SelectionMode == SelectionMode.MultiSimple)
                  {
                     // Toggle the selection state of this item.
                     SetSelected(index, !GetSelected(index)); 
                  }
                  else
                  {
                     // SelectionMode is One or None, so select the item. In the 
                     // case of None, the selection will be ignored.
                     SelectedIndex = index;
                  }
               }
               break;
            case ApplicationGesture.Up:
               InvalidateStrokesRect(e.Strokes);
               SendKeys.Send("{PGUP}");
               break;
            default:
               break;
         }
      }
    }
    catch (Exception ex)
    {
        // Provide a couple of methods for communicating the exception.
        // These could be replaced by any appropriate method of error
        // reporting and/or logging.
        System.Diagnostics.Debug.WriteLine(ex.ToString());
        System.Diagnostics.Debug.Assert(false, ex.ToString());
    }
}

Notice that the inkCollector_Gesture method checks InvokeRequired and marshals InkCollectorGestureEventHandler to the UI thread if required. This is because the Recognition event fires on a background thread when produced from a call to BackgroundRecognize. An event that is fired on a background (non-UI) thread like this may cause unpredictable behavior when accessing user interface objects, so the control uses Invoke to handle the event on the UI thread. For more information about threading considerations and recognition, see Mobile Ink Jots 4: Writing Solid Tablet PC Applications in the MSDN Library.

If InkListBox does not recognize the first stroke as a gesture, the Stroke event fires and is handled by the inkCollector_Stroke method. The Stroke object is added to the RecognizerContext object's Strokes collection and a timer and background recognition start to await the next stroke. Also, the CollectionMode value changes from InkAndGesture to InkOnly to prevent any future strokes from being interpreted as gestures. The following code shows how the InkListBox handles incoming strokes.

//
// inkCollector_Stroke
//
// Occurs when the user finishes drawing a new stroke on any tablet.
// This adds the Stroke to the RecognizerContext's Strokes collection, 
// to change the CollectionMode to InkOnly, and to restart the timer.
// This event does not fire if a gesture is recognized.
//
// Parameters:
//  sender - The source InkCollector object for this event.
//  e - The InkCollectorStrokeEventArgs object that contains the event data.
//
private void inkCollector_Stroke(object sender, InkCollectorStrokeEventArgs e)
{
    // This event can be fired on a background (non-UI) thread, 
    // which can cause unpredictable behavior when accessing UI objects. 
    // Use the Invoke method if necessary to handle the event on the UI thread.
    if (InvokeRequired)
    {
        Invoke(new InkCollectorStrokeEventHandler(inkCollector_Stroke),
            new object[] { sender, e });
        return;
    }

    // Prevent exceptions from propagating back to the sender -- the sender could ignore them, 
    // making it much harder to debug associated problems, and other event subscribers could 
    // be prevented from being called, leaving the system in an unanticipated state.
    try
    {
        recognizerContext.StopBackgroundRecognition();

        // Add the stroke to the RecognizerContext and switch to ink-only collection so we
        // don't misrecognize parts of letters as gestures.
        recognizerContext.Strokes.Add(e.Stroke);
        inkCollector.CollectionMode = CollectionMode.InkOnly;

        // Restart the timer and the background recognition.
        timer.Start();
        recognizerContext.BackgroundRecognize();
    }
    catch (Exception ex)
    {
        // Provide a couple of methods for communicating the exception.
        // These could be replaced by any appropriate method of error
        // reporting and/or logging.
        System.Diagnostics.Debug.WriteLine(ex.ToString());
        System.Diagnostics.Debug.Assert(false, ex.ToString());
    }
}

When the timer tick occurs, InkListBox presumes that the user has finished writing, and goes on to:

  • Stop background recognition.
  • Clear and erase the existing strokes.
  • Set CollectionMode back to InkAndGesture so that the next Stroke may be interpreted as a gesture, if necessary.
// 
// timer_Tick
//
// Occurs when the specified timer interval has elapsed and the timer is enabled.
// This event is used to end the current background recognition, clear 
// the existing Strokes, and then restart background recognition.
//
// Parameters:
//  sender - The source Timer object for this event.
//  e - The EventArgs object that contains the event data.
//
private void timer_Tick(object sender, EventArgs e)
{
   // Stop the timer and background recognition.
   timer.Stop();
   recognizerContext.StopBackgroundRecognition();

   // Remember InkEnabled state for restoration.
   bool wasInkEnabled = InkEnabled;
   SetInkEnabled(false);

   // Discard and erase all strokes.
   recognizerContext.Strokes.Clear();
   inkCollector.Ink.DeleteStrokes();
   Invalidate();

   // Restart background recognition.
   inkCollector.CollectionMode = CollectionMode.InkAndGesture;

   // Restore the previous state of InkEnabled.
   InkEnabled = wasInkEnabled;
}

When the RecognizerContext recognizes the user's strokes as text, it fires the Recognition event, which is handled by the recognizerContext_Recognition method. This event matches the recognized text to an item in the list box and selects the item, as shown in the following code.

// 
// recognizerContext_Recognition
//
// Occurs when the RecognizerContext object has generated results from the BackgroundRecognize method.
// This event is used to select the list box item, if any, that matches the recognized word.
//
// Parameters:
//  sender - The source RecognizerContext object for this event.
//  e - The RecognizerContextRecognitionEventArgs object that contains the event data.
//
private void recognizerContext_Recognition(object sender, RecognizerContextRecognitionEventArgs e)
{
    // This event can be fired on a background (non-UI) thread, 
    // which can cause unpredictable behavior when accessing UI objects. 
    // Use the Invoke method if necessary to handle the event on the UI thread.
    if (InvokeRequired)
    {
        Invoke(new RecognizerContextRecognitionEventHandler(recognizerContext_Recognition),
            new object[] { sender, e });
        return;
    }
    
    // Prevent exceptions from propagating back to the sender -- the sender could ignore them, 
    // making it much harder to debug associated problems, and other event subscribers could 
    // be prevented from being called, leaving the system in an unanticipated state.
    try
    {
        if (e.RecognitionStatus == RecognitionStatus.NoError)
        {
            int index = FindString(e.Text);
            if (index != -1)
            {
                SelectedIndex = index;
                Invalidate();
            }
        }
    }
    catch (Exception ex)
    {
        // Provide a couple of methods for communicating the exception.
        // These could be replaced by any appropriate method of error
        // reporting and/or logging.
        System.Diagnostics.Debug.WriteLine(ex.ToString());
        System.Diagnostics.Debug.Assert(false, ex.ToString());
    }
}

Modifying List Box Items

When a user modifies the collection of list box items (items are added to InkListBox or removed from it), the RecognizerContext's word list must be correspondingly updated. Also, the word list must be re-associated with the RecognizerContext, otherwise it does not detect the changes to the word list.

To detect list item changes, the control's WndProc method is overridden to catch any Windows messages related to the list box items collection:

/// <summary>
/// Overrides Control.WndProc so that the word list can be 
/// properly updated when messages are detected that alter the listbox item collection.
/// </summary>
/// <param name="m">The Message to process.</param>
/// <remarks>
/// The word list is updated in the addItemsTimer's Tick event instead of in this method 
/// because many messages can be received even when only one call was used to add an array 
/// of items to the listbox.  This ensures that the word list is only updated once,
/// which can provide a performance improvement for large lists.
/// 
/// The SecurityPermission attribute is used to prevent a possible security 
/// problem report by FxCop.
/// </remarks>
[SecurityPermission(SecurityAction.LinkDemand, 
   Flags=SecurityPermissionFlag.UnmanagedCode)]
protected override void WndProc(ref Message m)
{
   // Call the base class method first to process the messages 
   // before updating the word list.
   base.WndProc(ref m);

   // Don't check messages if in design mode, or if there are no recognizers.
   if (DesignMode || !recognizersInstalled)
   {
      return;
   }

   // Watch for ListBox messages which may alter content.
   const int LB_ADDSTRING = 0x0180;
   const int LB_INSERTSTRING = 0x0181;
   const int LB_DELETESTRING = 0x0182;
   const int LB_RESETCONTENT = 0x0184;
   const int LB_SETITEMDATA = 0x019A;
   const int LB_MULTIPLEADDSTRING = 0x01B1;

   switch (m.Msg)
   {
      case LB_ADDSTRING:
      case LB_INSERTSTRING:
      case LB_DELETESTRING:
      case LB_RESETCONTENT:
      case LB_SETITEMDATA:
      case LB_MULTIPLEADDSTRING:
         // Start or restart the timer.  The word list will be updated 
         // in the timer's Tick event handler.
         addItemsTimer.Start();
         break;
      default:
         // The message didn't affect the item list, so ignore it.
         break;
   }
}

The WndProc method does not actually rebuild the word list; instead, the method starts the addItemsTimer. This is done to prevent rebuilding the word list repeatedly when multiple items are added to the list box. The use of the timer enables users to add multiple items to the list. When this activity has completed and the timer's tick occurs, then the word list is rebuilt in aggregate, as shown in the following code sample:

// 
// addItemsTimer_Tick
//
// Occurs when the specified timer interval has elapsed and the timer is enabled.
// This event is used to synchronize the RecognizerContext.WordList with the 
// ListBox's Items.  See the WndProc method for more information on why 
// these actions are in this event handler.
//
// Parameters:
//  sender - The source Timer object for this event.
//  e - The EventArgs object that contains the event data.
//
private void addItemsTimer_Tick(object sender, EventArgs e)
{
   addItemsTimer.Stop();

   // Reset the WordList.  This must be done in whole, not 
   // incrementally, because changes to word lists are not picked 
   // up once they're assigned to a RecognizerContext.

   // The Strokes collection must be set to null before changing
   // the word list.
   Strokes strokes = recognizerContext.Strokes;
   recognizerContext.Strokes = null;

   WordList wl = new WordList();
   foreach (object item in Items)
   {
      wl.Add(item.ToString());
   }
   recognizerContext.WordList = wl;

   // Restore the Strokes collection.
   recognizerContext.Strokes = strokes;
}

Extending InkListBox

There are a number of ways that you can extend InkListBox to provide customized functionality for your enterprise or your unique application scenarios. Some suggested ways of extending the control are:

  • Trap additional gestures. You could enable recognition of other gestures that have specific meanings in your applications, for example a gesture to clear all items in a multi-select list.
  • Change default behavior. Your implementation of InkListBox could provide different default property values by modifying the constant declarations. For example, the default recognition timeout can be changed by changing the constant RecognitionTimeoutDefault. You can also easily add additional custom properties.
  • Enhance the inking experience. You could enable the user to modify the size and color of the ink strokes, and even preserve the strokes in a custom property for storage as part of an application's data set.

Conclusion

Customers' need for entering business data rapidly into custom applications provides unique challenges for developers, especially when the users are mobilized and not working on a desktop. This article demonstrates how to improve user productivity by combining the power of ink-enabled applications on Tablet PC with the rich user experience and scalability provided by Windows Forms controls.

By leveraging the Tablet PC Platform SDK together with the .NET Framework, you can provide improved data-entry experiences that best match the abilities and work environment of your application's users.

Biography

Leszynski Group is a solution provider and independent software developer specializing in development tools, Tablet PC applications, and mobilized databases. Leszynski Group is a regular contributor of MSDN content, and the creator of popular utilities such as the Snipping Tool and Physics Illustrator.