Filtering Stylus Input by Using a RealTimeStylus Plug-in on a Tablet PC

Article
10/16/2006

Jonathan Fingold
Microsoft Corporation

August 2004

Applies to:
Microsoft® Tablet PC Platform SDK
StylusInput APIs

Summary: This article describes how to extend the RealTimeStylus to limit ink collection to an arbitrary region. These descriptions and C# examples use the Microsoft Windows® XP Tablet PC Edition Development Kit version 1.7. Readers should be familiar with the Microsoft Tablet PC Platform SDK and managed code. In particular, readers should be familiar with Accessing and Manipulating Pen Input in the Tablet PC SDK 1.7. (12 printed pages)

Click here to download the sample code.

Introduction
Filter Plug-in
Ink-Collector Wrapper Plug-in
Custom Dynamic-Renderer Plug-in
The Sample Application
Other Considerations

Introduction

This article describes the sample C# project, which uses RealTimeStylus plug-ins to filter input from the stylus and to collect ink only within an arbitrarily shaped region. Enabling the user to ink in a multiple page view is one scenario in which filtering the stylus input may be useful.

This sample defines and uses the following plug-ins.

Filter plug-in: A synchronous plug-in that analyzes stylus input and adds custom data to the RealTimeStylus queue. The filter plug-in in this sample signals to the ink-collector wrapper plug-in when the stylus is within the filtering region.
Custom dynamic-renderer plug-in: A synchronous plug-in that renders ink as it is being collected. The custom dynamic-renderer in this sample renders ink within the filtering region.
Ink-collector wrapper plug-in: An asynchronous plug-in that acts as an interface between the RealTimeStylus and an instance of the ink-collector plug-in. The ink-collector wrapper plug-in in this sample uses the data created by the filter plug-in and the RealTimeStylus to simulate a stylus that is only down while it is within the filtering region.
Ink-collector plug-in: An asynchronous plug-in that converts stylus input to ink. The ink-collector plug-in in this sample is managed by the ink-collector wrapper plug-in. The ink-collector plug-in could be used by itself with a RealTimeStylus to create a standard ink-collecting application that does not use a filtering region.

In addition, the sample provides a user interface for enabling or disabling the dynamic rendering and for enabling or disabling the filtering. The sample contains two projects, FilteringSampleApp and InkCollectorPlugin. The FilteringSampleApp project contains the form, the filter plug-in, the custom dynamic-renderer plug-in, and the ink-collector wrapper plug-in. The InkCollectorPlugin project contains the ink-collector plug-in. The InkCollectorPlugin project defines the RealTimeStylusPlugins namespace, which is referenced in the FilteringSampleApp project. Both projects use the Microsoft.Ink, Microsoft.StylusInput, and Microsoft.StylusInput.PluginData namespaces.

This article does not discuss:

How to create a RealTimeStylus application.
How to perform ink collection by using the RealTimeStylus.

For more information about the RealTimeStylus class, see Accessing and Manipulating Pen Input, RealTimeStylus Plug-in Sample, and RealTimeStylus Ink Collection Sample in the Tablet PC SDK 1.7.

Filter Plug-in

This section describes the design and implementation of the filter plug-in in the sample.

The filter plug-in is designed to work in conjunction with the ink-collector wrapper plug-in. The following list describes the interaction between the filter plug-in and the ink-collector wrapper plug-in.

The filter plug-in analyses the stylus data stream and adds custom stylus data to the stream to signal to the ink-collector wrapper plug-in when the stylus is within the filtering region. In addition, when the filter plug-in receives a Packets notification that contains packets on both sides of the filtering region, then the filter plug-in adds a PacketsData object to the stylus data stream for the packets that are inside the filtering region. Also, the filter plug-in adds FilterPluginEnabledData and FilterPluginDisabledData objects to the queue when it is enabled or disabled.
While the filter plug-in is enabled, the ink-collector wrapper forwards the StylusDownData, PacketsData, and StylusUpData objects added by the filter plug-in to a private ink-collector plug-in that it maintains. Also, while the stylus is within the filtering region, the ink-collector wrapper plug-in forwards PacketsData objects it receives from the RealTimeStylus to the private ink-collector plug-in.
While the filter plug-in is disabled, the ink-collector wrapper forwards all StylusDownData, PacketsData, and StylusUpData objects it receives from the RealTimeStylus to the private ink-collector plug-in.

The filter plug-in implements the IStylusSyncPlugin interface and performs the following functions.

Analyzes the stylus data stream to determine when the stylus crosses the boundary of the filtering region.
Adds custom stylus data to the RealTimeStylus queue to signal when the stylus moves into or out of the filtering region.

The following table describes the condition under which the filter plug-in adds each type of data to the queue.

Data	Condition
FilterPluginEnabled	The filter plug-in becomes enabled, or the RealTimeStylus becomes enabled while the filter plug-in is already enabled.
FilterPluginDisabled	The filter plug-in becomes disabled.
StylusDownData	A StylusDown occurs within the filtering region, or the stylus enters the filtering region.
StylusUpData	A StylusUp occurs within the filtering region, or the stylus leaves the filtering region.
PacketsData	A Packets notification crosses the filtering region's boundary.
object[]	When custom data for multiple notifications can be added to the queue at the same time, the custom data is added as an array.

To minimize the amount of custom data generated by the filter plug-in, Packets notifications are only added when the original Packets notification contains a set of packets that span the boundary of the filtering region.

For details of the implementation of the filter plug-in, see the source code for the FilterPlugin class in the FilterPlug.cs file.

Preliminaries

Identification of the Filter Plug-in

The filter plug-in defines a static field, FilterPluginGuid, which contains a globally unique identifier (GUID) to identify the custom data it adds to the queue.

Enabling and Disabling the Filter Plug-in

The filter plug-in provides the public Enabled property to allow the form to update the enabled state of the plug-in. While the RealTimeStylus is enabled, the filter plug-in notifies the ink-collector wrapper plug-in of its enabled state by using the FilterPluginEnabledData and FilterPluginDisabledData classes defined in the FilterPlugin.cs file.

Tracking the RealTimeStylus to which the Filter Plug-in Is Attached

The filter plug-in subscribes to the RealTimeStylusEnabled and RealTimeStylusDisabled notifications in order to:

Maintain its internal state.
Generate FilterPluginEnabled notifications as necessary.

In addition, the filter plug-in is not designed to handle input from more than one RealTimeStylus at a time, and it throws an InvalidOperationException if it receives a RealTimeStylusEnabled notification from a second RealTimeStylus without the first one sending a RealTimeStylusDisabled notification.

Managing the Filtering Region

The filtering region is set by the application through the FilteringRegion property of the filter plug-in. In this sample the filter plug-in does not allow the filtering region to change while the filter plug-in is enabled. The property can be set to null, but the filter plug-in can not be enabled while the filtering region is undefined.

The Graphics Object

The filter plug-in maintains a reference to a Graphics object, in its theGraphics field, for the control that is collecting the stylus input. This Graphics object is used in creating the transformation matrices for the filtering region.

Converting between Ink-Space and Pixel Coordinates

The stylus data is in ink-space coordinates, and the filter plug-in stores the filtering region internally in ink-space coordinates. However, the property that exposes the filtering region is in pixel coordinates.

Two matrices, in the inkSpaceToPixelMatrix and pixelToInkSpaceMatrix fields, are declared that help to convert the filtering region to and from pixel and ink-space coordinates. These matrices are used when getting or setting the FilteringRegion property.

Adding Custom Stylus Data

Conditions

The filter plug-in adds StylusUpData, PacketsData, and StylusDownData to the RealTimeStylus queue. The following table describes the conditions under which data is added to the queue and what data is added.

Notification	Condition	Action
StylusDown	The StylusDown notification originates within the filtering region.	The filter plug-in adds the StylusDownData object to the Output queue.
StylusUp	The StylusUp notification originates within the filtering region.	The filter plug-in adds the StylusUpData object to the Output queue.
Packets or StylusUp	The line between the previous location of the stylus and the Packets or StylusUp notification crosses into the filtering region.	The filter plug-in adds a StylusDownData object at the intersection of the line and the filtering region boundary to the OutputImmediate queue.
Packets or StylusUp	The line between the previous location of the stylus and the Packets or StylusUp notification crosses out of the filtering region.	The filter plug-in adds a StylusUpData object at the intersection of the line and the filtering region boundary to the OutputImmediate queue.
Packets	The Packets notification crosses into the filtering region.	The filter plug-in caches a StylusDownData object at the intersection of the line where the packets enter the filtering region and the filtering region boundary.
Packets	The Packets notification contains packets within the filtering region.	The filter plug-in caches packet data for packets within the filtering region.
Packets	The Packets notification crosses out of the filtering region.	The filter plug-in adds a StylusDownData object if one is cached, adds a PacketsData object for the cached packets inside the region, and adds a StylusUpData object to the OutputImmediate queue. The filter plug-in then clears the cached data.
Packets	The last packet is inside the filtering region and a StylusDownData object is cached.	The filter plug-in adds the cached StylusDownData object and adds a PacketsData object for the cached packets inside the region to the Output queue.

Timing Considerations

The position of the StylusDownData and StylusUpData objects added by the filter plug-in in relation to the PacketsData generated by the RealTimeStylus is important because the ink-collector wrapper plug-in accepts all Packets notifications occurring between the custom StylusDown and StylusUp notifications.

The filter plug-in adds StylusUpData objects to the RealTimeStylus OutputImmediate queue so that they are placed on the queue before the PacketsData that spans the filtering region's boundary.
The filter plug-in adds StylusDownData to the RealTimeStylus Output queue so that they are placed on the queue after the PacketsData that spans the filtering region's boundary.
The filter plug-in caches some data while it processes Packets notifications so that it can package custom data together and add the data to the output queue in the correct order.

Implementation

For details of the implementation of the filter plug-in, see the source code for the StylusDown, StylusUp, and Packets methods of the FilterPlugin class in the FilterPlug.cs file. The StylusUp and Packets methods use the FindIntersection helper method which finds the intersection of a line and the boundary of the filtering region. The FindIntersection method uses successive bisection of a line segment to estimate the intersection point. The Packets method uses three additional helper methods, ParsePacketsData, ProcessFirstPacket, and ProcessNextPacket, which handle portions of the analysis of the packets data.

Additionally, the previousLocation and outsideRegion instance fields are used to track the previous state of the stylus.

Refinements

The implementation of the filter plug-in is not complete. Specifically, two adjacent packets can be far enough apart that they are both inside or both outside the filtering region, but the line between them crosses the region's boundary. In this case, the filter plug-in does not detect the crossing of the region's boundary.

One solution to this problem would be to specify a minimum feature size the filter plug-in can detect. If two successive packets are more than this distance apart, then divide the line segment between them into sub-segments no longer than the feature size and check for transitions across each of the sub-segments.

Ink-Collector Wrapper Plug-in

The ink-collector wrapper plug-in implements the IStylusAsyncPlugin interface and maintains a private ink-collector plug-in. The ink-collector wrapper plug-in checks for data from the filter plug-in and acts as the RealTimeStylus stream to its private ink-collector plug-in. For details of the implementation, see the source code for the InkCollectorWrapper class in the InkCollectorWrapper.cs file and for the InkCollectorPlugin class in the InkCollectorPlugin.cs file

The ink-collector wrapper plug-in monitors whether the filter plug-in is enabled through the FilterPluginEnabledData and FilterPluginDisabledData objects the filter plug-in adds to the RealTimeStylus queue. If the filter is enabled, the ink-collector wrapper plug-in forwards the following data.

StylusDownData, PacketsData, and StylusUpData objects generated by the filter plug-in as StylusDown, Packets, and StylusUp notifications.
Packets notifications generated by the RealTimeStylus that occur between filter plug-in StylusDownData and StylusUpData objects.

If the filter plug-in is disabled, the ink-collector wrapper plug-in forwards to its private ink-collector plug-in all StylusDown, Packets, and StylusUp notifications from the RealTimeStylus.

In its constructor, the ink-collector wrapper plug-in creates the following objects.

A Graphics object for the control from which the ink-collector wrapper plug-in receives stylus data and to which the ink-collector wrapper plug-in draws the collected strokes.
The ink-collector plug-in, which creates and stores the ink.

When a stroke is completed, the ink-collector wrapper plug-in calls the Refresh method of the ink-collector plug-in to draw the collected ink on the form.

Implementation

For details of the implementation of the ink-collector wrapper plug-in, see the source code for the StylusDown, StylusUp, Packets, and CustomStylusDataAdded methods of the InkCollectorWrapper class in the InkCollectorWrapper.cs file. The CustomStylusDataAdded method uses the ProcessCustomData helper method.

The ink-collector plug-in is defined in the InkCollectorPlugin project, which generates a .dll file that is referenced by the filtering sample application. The ink-collector plug-in subscribes to the StylusDown, Packets, and StylusUp notifications, and creates strokes. It uses the design of the ink-collector plug-in from the RealTimeStylus Ink Collection Sample in the Tablet PC SDK 1.7.

Custom Dynamic-Renderer Plug-in

The custom dynamic-renderer plug-in implements the IStylusSyncPlugin interface and renders a stroke as it is being drawn. The custom dynamic-renderer plug-in only renders the stroke within the filtering region.

For details of the implementation of the custom dynamic-renderer plug-in, see the source code for the CustomDynamicRendererPlugin class in the CustomDynamicRenderer.cs file.

The custom dynamic-renderer plug-in defines four public properties and two public methods.

The ClippingRegion property gets or sets the Clip property of the Graphics object to which the plug-in renders. This property allows the custom dynamic-renderer plug-in to use the same rendering algorithm whether or not a clipping region is set.
The Enabled property gets or sets the enabled state of the plug-in, which controls whether dynamic rendering is on or off.
The Color property gets or sets the color of the pen used to render the stroke.
The Width property gets or sets the width of the pen used to render the stroke. The Width property is in HIMETRIC units (.01mm), but the width of the Pen object is in pixel units.
The Clear method clears all stored information about the current stroke.
The Refresh method redraws the current stroke that is being collected.

The custom dynamic-renderer plug-in also subscribes to the StylusDown, Packets, and StylusUp notifications. The plug-in uses the data from these notifications to render each stroke as it is draw and to determine when the stroke is finished.

The ConvertFromInkSpaceToPixels helper method converts a Point structure from the ink space coordinates of the RealTimeStylus notifications to the pixel coordinates of the Graphics object for the control.

The Sample Application

The FilteringSampleForm class is a form and implements the IStylusAsyncPlugin interface. The form collects ink using the filter plug-in and the ink-collector wrapper plug-in. The form also renders the stroke as it is being collected using the custom dynamic-renderer plug-in. The form sets the custom dynamic-renderer plug-in's Color property to red to distinguish between dynamic rendering and the static rendering of the ink-collector plug-in.

For details of the implementation of the sample application, see the source code for the FilteringSampleForm class in the FilteringSampleForm.cs file.

The form's main menu provides the following functionality.

The File menu's Clear item removes collected ink from the form.
The File menu's Exit item closes the form.
The DynamicRenderer menu's Enable item toggles the enabled state of the custom dynamic-renderer plug-in.
The Filter Plug-in menu's Enable item toggles the enabled state of the filter plug-in.
The Filter Plug-in menu's Show InputRegion item toggles the display of the filtering region, which is drawn in light gray.

The custom dynamic-renderer plug-in renders strokes within its clipping region. The form keeps the custom dynamic-renderer plug-in's clipping region synchronized with the state of the filter plug-in and its filtering region.

While the filter plug-in is enabled, the ink-collector wrapper plug-in only collects ink within the filtering region. While the filter plug-in is disabled, the ink-collector wrapper plug-in collects ink anywhere on the form.

The form handles three events in addition to the click events for the menu items.

The FilterPluginForm_Load method handles the form's Load event. This method does the following.
- Creates the RealTimeStylus and the filter, custom dynamic-renderer, and ink-collector wrapper plug-ins.
- Attaches the plug-ins to the RealTimeStylus.
- Enables the plug-ins and the RealTimeStylus.
- Initializes instance fields and the checked state of some of the menu items.
The FilterPluginForm_Closed method handles the form's Closed event. This method disables and disposes of the RealTimeStylus, disposes of the filtering region, and releases the references to the plug-ins.
The FilterPluginForm_Paint method handles the form's Paint event. This method draws the filtering region and calls the Refresh methods of the custom dynamic-renderer and ink-collector wrapper plug-ins.

The form subscribes to the Error notification, and displays a message when the RealTimeStylus catches an exception thrown by one of the plug-ins.

The following diagram illustrates the flow of data in this sample within the RealTimeStylus and the attached plug-ins. For more information about using the RealTimeStylus class, see Accessing and Manipulating Pen Input in the Tablet PC SDK 1.7.

Figure 1. Data flow in the sample application.

In figure 1 the circles lettered "A" and "B" represent tablet pen data that has already been added to the RealTimeStylus object's output queue and that has not yet been sent to the asynchronous plug-in collection. The circle lettered "C" represents the tablet pen data that the RealTimeStylus object is currently processing. It is sent to the synchronous plug-in collection and placed on the output queue. The empty circle represents the position in the output queue where future tablet pen data is added. The circles lettered "FP" represent custom stylus data objects added by the filter plug-in to the RealTimeStylus object's output queue in response to the data labeled "C". In addition, this diagram shows that the ink-collector wrapper plug-in forwards data from the filter plug-in and RealTimeStylus to its private ink-collector plug-in.

In the AssemblyInfo.cs file for both projects, the assembly requests the UIPermissionWindow.SafeTopLevelWindows permission for the RealTimeStylus. In the AssemblyInfo.cs file for the InkCollectorPlugin project, the assembly also requests SecurityPermissionFlag.UnmanagedCode permission for the Renderer object. For more information about security and trust considerations for Tablet PC applications, see Security And Trust in the Tablet PC SDK 1.7.

Other Considerations

This article describes a rudimentary filter plug-in, ink-collector wrapper plug-in, and application. The following list describes some elements to consider when creating a stylus input filtering application.

Boundary Detection

If the stylus is moving quickly, the boundary detection of the filter plug-in can miss features of the bounding region. To reliably detect transitions across the boundary of the filtering region, add a distance calculation to successive packets within a stroke. If any packets are farther apart than the minimum feature size of the region you want to detect, then subdivide the line between the two points into segments no greater than the minimum feature size, and perform boundary transition checks for each of the segments. The minimum feature size you want to detect may depend upon the application and the type of filtering region you use, as well as performance considerations.

The following diagrams illustrate the boundary detection issues present in the filter plug-in.

Figure 2. Issues in boundary detection

In these diagrams, the green shaded areas represent the interior of the filtering region. The numbered circles represent individual packets that the RealTimeStylus reports to its plug-ins. The lettered circles represent the intersection points that the filter plug-in detects. And, the unlabeled circles with the broken border represent subdivision points of the original packet data.

In diagram A both packet "1" and packet "2" are inside of the filtering region, and the filter plug-in does not detect the transition of the stroke out of and back into the filtering region.
In diagram B packets "1", "2", and "3" are outside of the filtering region, and the filter plug-in does not detect the transitions of the stroke into and out of the filtering region.
In diagrams A' and B', with the addition of points between the packets in diagrams A and B respectively, the filter plug-in now detects the transitions of the stroke across the boundary of the filtering region.

Interpolation Method

This sample uses linear interpolation to find the intersection of the stroke and the boundary of the filtering region. For some applications, using a spline instead of a straight line to determine the intersection may be more appropriate.

Pressure Data

The filter plug-in does not interpolate pressure data when it adds StylusDownData and StylusUpData objects at the intersection points of the stroke and the boundary of the filtering region. The FindIntersection method of the FilterPlugin returns the fraction of the distance from the first point to the second point where the intersection occurs. For example, a return value of 0.5 indicates that the intersection occurs exactly half way between the two points. This value can be used to linearly interpolate the pressure data for the intersection point.

Units Conversion

Both the filter plug-in and the custom dynamic-renderer plug-in convert from HIMETRIC units (.01mm) to pixel units using the dots per inch (DPI) resolution of the Graphics object and the conversion factor of 1 inch = 2540 HIMETRIC units. Performing this conversion in your application is more efficient than using the Renderer object's PixelToInkSpace and InkSpaceToPixel methods, as the Renderer is tuned for working with strokes.

Dynamic Rendering

The custom dynamic-renderer plug-in could implement a DrawingAttributes property instead of Width and Color properties. This property can then be synchronized with the DrawingAttributes applied to the ink-collector plug-in.