Export (0) Print
Expand All
Add Support for Digital Ink to Your Windows Applications
Power to the Pen: The Pen is Mightier with GDI+ and the Tablet PC Real-Time Stylus
Expand Minimize

Developing Ink-Driven Applications with OrangeGuava Desktop

 

Tom Clarkson
OrangeGuava

December 2005

Applies to
   Microsoft Tablet PC Platform SDK
   OrangeGuava Desktop

Summary: Introduces an ink-driven application model, using the OrangeGuava Desktop SDK. (11 printed pages)

Contents

Introduction
Benefits of the Ink-Driven Approach
The Platform
Making a First Application
Doing Something Useful
Deployment
Making a Real-World Application

Introduction

While the Tablet PC platform enables you to use ink in standard Windows Forms Applications, the new form factor also makes it possible to use a computer in ways that are closer to how people naturally perform tasks.

The ink-driven application model is one such approach. Users write whatever they choose to on a permanently available input area, and the data is sent to the appropriate application for processing, rather than taking the time to find and start an application that does what they want.

This article discusses the benefits of an ink-driven approach, the platform diagram, and how to build an ink-driven application, using a simple currency converter as an example. Many of the examples are based on the model for the application I created to ink-enable the Windows desktop, an application called OrangeGuava Desktop.

Benefits of the Ink-Driven Approach

The ink-driven model simplifies things for both the developer and the end user by offering the following:

  • The user can quickly interact with applications on the desktop without being distracted by navigating a potentially lengthy start menu.
  • The developer can leave out a lot of user interface code and design, as most of the front end is handled by the input application.

The platform works well for small utilities such as the currency converter example in this article. When users spend little time using applications, the cost of finding and starting them is proportionately higher. However, larger applications can also benefit from this approach, in that they can often be viewed as collections of smaller applications.

The Platform

Figure 1. Basic structure of an ink-driven application

There are three main components in the Ink-driven application model;

Input Application

An ink-driven application makes use of a generic input area, which provides all the standard ink handling functions (erasing, saving to the Clipboard, and others) and can be used by multiple applications. Each application does not require a separate input window.

Currently, only OrangeGuava Desktop itself provides this service, but a future version of the OrangeGuava SDK will enable other applications to easily offer this functionality.

XML Configuration

The platform uses an XML file to tell the input application about the destination application. As well as setting the paths and method names to be called, the XML file includes information on what handwritten information is likely to be able to be processed by the application.

Destination Application

The application that actually uses the ink data listens for messages from the input application rather than displaying its own input form. The code to be placed in an application to enable ink-driven operation is fairly simple. The code is basically an initialization call and a static method that accepts the strokes and recognition information delivered by the input application.

Making a First Application

This section details the steps you need to take to implement an ink-driven solution.

Setting up the Project

There are two options for setting up your application:

  • As a class library running within the input application process. This is good for very simple applications that run entirely within a single assembly or only access globally-accessible files. I do not discuss this option in detail, as it is effectively a cut-down version of an executable ink-driven application.
  • As an application. Everything is the same as for a regular application, except that the user interface is not visible.

Although the currency converter example is simple enough to run well as a library, create it as an application because that option enables the most complete functionality.

There are two assemblies that you need to add to your project. Both are included in the merge module provided with the OrangeGuava Desktop SDK, which places them in the global assembly cache.

  • OrangeGuava.Formalize—Defines the class AbstractFormalizeData, which encapsulates the ink and recognition information from the input application.
  • OrangeGuava.Formalize.Listener—Includes code to listen for messages from the input application. This one is not required if your application is created as a class library.

You will also need to add a reference to the Tablet PC API.

The Main Method

      static void Main() 
      {
         Form1 form1 = new Form1();
         OrangeGuava.Formalize.FormalizeListener.Initialize("currencyconvert");
         Application.Run(form1);
      }

In the startup method of your application, call FormalizeListener.Initialize to begin listening for messages from the input application. This method also processes any messages already in the queue, so any initialization code (such as the form constructor) that needs to be run before messages are processed should be run before initializing the listener.

The string parameter currencyconvert is the name of the queue to which your application will be listening. The only requirement for this value is that it should uniquely identify your application and contain only characters valid in a file name.

For now, leave the application starting up as a standard Windows Form, mainly because that makes it easier to stop and start during development. Because the form doesn't do anything useful, the application should really start without showing any user interface, possibly using a system tray icon to give the user some control over what is running.

Checking Requirements

      public static bool Convert() 
      {
         return true;
      }

Add a method to determine whether or not the application can be used. This is the place to check if any required applications are installed. The input application calls this method during initialization, and if it returns false, your application is not be made available to the user.

The Ink Processing Method

      public static bool Convert(OrangeGuava.Formalize.AbstractFormalizeData fd) 
      {
         MessageBox.Show(fd.Strokes.Count.ToString());
         return true;
      }

Add a method to do the actual processing of the ink input. This method must have the same name as the requirement checking method, the only difference being that it takes an AbstractFormalizeData structure as a parameter. This structure contains the input in three forms:

  • A Strokes collection
  • A byte array containing the strokes in ISF format
  • The recognized and corrected text sorted into named fields.

Leave this method almost empty until it runs correctly.

The XML file

Now that you have an application with all the necessary parts to process ink input, the input application just needs to know where to find it. This is done with an XML file placed in either:

  • Common Files\OrangeGuava\Formalize (Available to all users)
  • Application Data\OrangeGuava\Formalize (Available only to the current user)
<formalizesettings>
  <structure guid="{EC8907C4-A481-4535-9F5C-0E44F684535E}">
<destination guid="{2C399AB8-9A29-49ab-8430-1C1AE2EA32AF}" 
assemblypath="[Output path here]\CurrencyConverter.exe"
classname="OrangeGuava.Formalize.Destinations.CurrencyConverter.Form1"
methodname="Convert" 
programfile="[Output path here]\CurrencyConverter.exe" 
title="Currency Converter"
description="An application to convert currencies"
programid="currencyconvert"
/>
  </structure>
</formalizesettings>

Once the correct output path has been added, the XML file contains the minimum amount of information needed to make your application visible to the input application. The GUID attribute of the structure element is the GUID for plain text, which means that the input application will not attempt to create a suitable data structure before sending the input.

The destination element is where your application is described. The first three attributes allow reflection to find the ink processing method.

The programfile attribute is the path to the program that needs to be running in order for messages to be received. For smaller applications this will almost always be the same as the assemblypath attribute.

The title and description attributes are used in the input application's user interface.

The programid attribute is the same unique queue name passed to the FormalizeListener initialization method.

Doing Something Useful

At this point you have an ink-driven application that works. Of course, all it does is accept random input and display a message box, which isn't particularly useful.

The following are two options for using the ink input in your application:

  • Read the original strokes data and handle the recognition manually. This is a good option if you are adapting a standard InkCollector and form application, because you can use the same code that usually processes ink from the form.
  • Have the input application handle the recognition and correction, providing your application with string data in named fields.

Instead of using the plain text structure, create a custom structure suitable for currency information: the amount and two currencies. To keep things simple and avoid any parsing in the destination application, put each field on a separate line and make all the fields required.

Figure 2. Sample input for the currency converter

This will require some additional information in the XML file:

<structure 
guid="{AA29AC1C-0786-4c31-82D5-4A74912135F2}" 
name="Currency Conversion" 
minlines="3"
maxlines="3" 
requirecount="3">

<field 
guid="Amount" 
title="Amount" 
requirescore="1" 
matcher="(0|1|2|3|4|5|6|7|8|9|\.)+" 
weightmatcher="2" 
factoid="(0|1|2|3|4|5|6|7|8|9|\.)+"
/>

<field 
guid="From" 
title="From" 
requirescore="1" 
matcher="(USD)|(NZD)|(THB)" 
weightmatcher="2" 
factoid="(USD)|(NZD)|(THB)" 
preferredindex="1" 
weightindex="1"
/>

<field 
guid="To" 
title="To" 
requirescore="1" 
matcher="(USD)|(NZD)|(THB)"
weightmatcher="2"    
factoid="(USD)|(NZD)|(THB)"  
preferredindex="2" 
weightindex="1" />

<destination ... />
</structure>

The structure element has been given a new GUID, so the currency converter no longer works from the plain text data structure. The purpose of the name attribute is obvious enough, and the remaining attributes define how the input application will determine if ink can be sent to the currency converter application. In this case, the selected ink must recognize exactly three lines of text, and three required fields need to be matched. In real-world use, minlines and requirecount would probably be 2, so that a default value could be used for one of the currencies, but the additional code required to let the user choose a default is beyond the scope of this article.

If the selected ink has the right number of lines, the three field elements become relevant. Each has a unique id and a title displayed to the user, along with some recognition settings.

The requirescore attribute specifies that the field contributes to the count of required fields in the structure if one of the lines matches that field.

The matcher and factoid attributes contain regular expressions specifying the expected format of the field. Although in this case they are the same, they are used slightly differently. The factoid attribute alters the recognition result and needs to specify the format for the full line, while the matcher attribute only searches for the expression within the line, ignoring anything else on the line.

The preferredindex attribute provides a way to distinguish between the to and from fields, in that the from field always appears first.

The weight attributes tell the system that the formatting of the fields (weightmatcher=2) is more important than their position (weightindex=1).

With appropriate fields defined, you should now be able to use the more structured data in the ink processing method:

      public static bool Convert(OrangeGuava.Formalize.AbstractFormalizeData fd) 
      {
         string amount = fd.GetValueString("Amount");
         string from = fd.GetValueString("From");
         string to = fd.GetValueString("To");

         IShellDispatch4 sc = (IShellDispatch4)new ShellClass();
sc.ShellExecute("http://www.xe.com/ucc/convert.cgi?From="+from+"&To="+to+"&Amount="+amount, null, null,null,null);

         fd.Strokes.Ink.DeleteStrokes(fd.Strokes);

         return true;
      }

The fields defined in the XML file are accessed by using the AbstractFormalizeData.GetValueString method. Although they are often the same, the parameter needs to match the id attribute of the field rather than the title. With all the fields marked as required, you can be reasonably certain that the correct values will be returned, and send the parameters off to a Web site so as to get something functional without writing any more code.

As well as processing the information that has been provided as recognized text, you can manipulate the original Strokes collection. This example shows the most common usage, deleting the visible strokes once the operation is complete so that the user's desktop is not cluttered. There are a lot of other possibilities, including getting input from a custom recognizer, changing the drawing attributes of the strokes, or returning information as new strokes (as is done in xThink Calculator).

Deployment

Your application can be deployed by using Windows installer with only a couple of additions that are specific to the ink-driven functionality.

The OrangeGuava Desktop SDK includes a redistributable merge module, OrangeGuava.Formalize.msm, that places the necessary assemblies in the global assembly cache. By including the merge module you ensure that your application runs without errors, whether or not a suitable input application is installed.

Because the application's install path can change, hard-coded paths cannot be used in the XML file. Instead, attributes must be added to describe a registry value that contains the path to the application.

<destination 
guid="{2C399AB8-9A29-49ab-8430-1C1AE2EA32AF}"
classname="OrangeGuava.Formalize.Destinations.CurrencyConverter.Form1"
assemblypath="CurrencyConverter.exe" 
methodname="Convert"
title="Currency Converter" 
description="Convert"
programid="currencyconvert" 
programfile="CurrencyConverter.exe"
programpathrkey="SOFTWARE\OrangeGuava\Formalize\CurrencyConverter"
programpathrval="Directory" 
/> 

The assemblypath and programfile attributes have been changed to include only the file names, and the attributes programpathrkey and programpathrval hold the registry key and value names.

The easiest way to provide a suitable registry key is to set one up in the registry section of the install project, using [TARGETDIR] as the value. The key can be placed in either HKEY_LOCAL_MACHINE or HKEY_CURRENT_USER.

Making a Real-World Application

This program is a simple example. A number of other features would be expected in a real-world application. Some things to consider when creating a more robust application are:

  • Sometimes the same application may be started either by the input application (hidden) or by the user accessing the Start menu (displaying a form). If your application can operate in both modes, you should include code in the startup method to check if another instance of the program is running and if so send a message to display the hidden instance rather than starting a new one. Allowing a second process to start is not only untidy, but it may result in messages from the input application being processed more than once.
  • If your application uses a significant amount of resources and is not in constant use, it may be worth adding a timer to shut down the application if it goes unused for a while, as the user may be unaware that the program is still running.

This article provides the necessary information to start building some useful ink-driven applications, and serves as a starting point for more innovative ways of building Tablet PC applications.

Go ahead and create something no-one was expecting!

Show:
© 2014 Microsoft