Mobile Ink Jots 2: All About Lonestar

Article
06/29/2006

Shawn A. Van Ness
Leszynski Group

July, 2004

Applies to:
Microsoft® Windows® XP Tablet PC Edition 2005
Microsoft Windows XP Tablet PC Development Kit 1.7

Summary: The release of Microsoft Windows XP Tablet PC Edition 2005, codenamed "Lonestar", offers many new features and enhancements. In this issue of Mobile Ink Jots, Shawn Van Ness describes the major innovations introduced both for the end user and the developer. (11 printed pages)

Introduction
What Is Lonestar?
The New Input Panel Experience
Enhancing Recognition with Context Tagging and the New SetInputScope API
Support for Ink on the Web
The New RealTimeStylus API Layer
SDK Versioning: Where Are We?
Conclusion

Introduction

Welcome to the second installment of the new Mobile Ink Jots column. David Hale began things last month with a tour of the various ISV partnership programs available for Tablet PC application developers. Starting this month, we're going to change the tack of this column and begin discussing Tablet PC development on a more technical level. But before we get too buried in the details, let's continue to consider the many reasons software developers have to be excited about the Tablet PC. I'm not referring to the $100,000 contest—David covered that last month. I'm talking about Windows XP Tablet PC Edition 2005—codenamed "Lonestar"—and its associated new SDK.

From the perspective of both end-users and developers, the Lonestar release offers a lot to be excited about:

A greatly enhanced user experience for handwriting recognition.
The ability for applications to give contextual hints to the recognizer.
Support for running Ink applications from partially-trusted sources (for example, Web pages).
A whole new API for high-performance, low level access to the digitizer's data stream.

Each of these areas is a topic worthy of a column all its own. Rest assured, we'll be addressing them individually, and in greater detail, in the upcoming months. This month I'll just give an overview of each of these topics, focusing on what you need to be prepared for, today, to ensure a smooth transition to the next generation Tablet PC platform, tomorrow.

What Is Lonestar?

First things first: let's be clear on what Lonestar is not. Lonestar is not Longhorn; despite the similar names, the two projects are not related in any meaningful way. Longhorn—the code name for the next major version of Windows—has gotten a lot of press recently, overshadowing Lonestar. That's unfortunate, because Lonestar will ship in a matter of weeks, not years. While Longhorn is still in the early stages of development, the Lonestar enhancements for Tablet PC will be delivered as part of Windows XP Service Pack 2 (SP2).

Of course, the Lonestar release will be accompanied by a new version of the Tablet PC SDK (Windows XP Tablet PC Development Kit, version 1.7) which, in addition to support for all the shiny new features of Lonestar, sports an excellent new set of documentation. More details on the new SDK, later in this article.

Reminder: the official name of Lonestar is Windows XP Tablet PC Edition 2005. But you don't have to wait for 2005, it's already here. Preview releases of Windows XP Service Pack 2 are available for public download, and the official release may be available by the time you read this. A word of warning is needed here: Windows XP Tablet PC Edition 2005 offers a vastly improved handwriting recognition experience. You'll quickly become addicted and want to upgrade every Tablet PC you own, but you should wait for the official release of SP2 before upgrading any Tablet PC which you rely on for day to day, production use. And, unless you plan on mandating that all your customers upgrade to SP2 before installing your product (not a bad idea, by any means) you might wish to keep some of your Tablet PCs running the original release for a while. You'll need them for testing.

The New Input Panel Experience

In last month's Tablet 101 column, Frank Gocinski gave a peek at the new Tablet PC Input Panel. This new Input Panel completely changes the look and feel of the Tablet PC. As you can see from the screenshot in his article, it pops up near the point of text insertion and grows to accommodate long strings of text. But a screenshot can only show you so much. What you can't see in that screenshot is the new experience for correction, enabled for western languages. As you write, the top recognition alternates appear, asynchronously, alongside your handwriting. This is shown in Figure 1.

Figure 1. Top recognition alternates automatically appear.

Tapping one of these words brings up another window, shown in Figure 2. This window allows you to select from a list of alternates or correct the recognition result, letter-by-letter.

Figure 2. The new correction experience.

With a bit of practice, this new user interface allows users to perform even complex correction tasks, like joining adjacent words together or splitting a word apart, typically with just a few taps or strokes of the pen. The correction UI even goes to great lengths to distinguish visually similar characters, such as the lowercase Latin "l" and the number "1", or the uppercase Latin "O" and the number "0".

It really is a qualitatively different experience, especially when combined with the power of recognition context (discussed later). I am a keyboard-fanatic software developer who doesn't use any feature of any application without memorizing its keystroke sequence (the AA batteries in my wireless optical mouse have lasted for years), and I can say that the new Input Panel experience finally makes the Tablet PC operating system fully usable with a pen. I will probably always type many times faster than I can write, but now I regularly sit on my couch and surf the web—my Tablet PC on my lap—without ever resorting to the keyboard.

Those of you who've been doing Tablet PC development for a while may be wondering, "What about the PenInputPanel"?

The PenInputPanel class is a platform feature introduced with version 1.5 of the Tablet PC Platform SDK. The PenInputPanel class provides a pop-up window for pen input that can be programmatically attached to any edit control, adding Tablet PC Input Panel functionality to existing controls. The PenInputPanel class still exists in the SDK, but its use is deprecated for new applications. To put a better spin on that: the PenInputPanel class was such a good idea that it was augmented and promoted to a built-in feature of the entire operating system, on by default for all edit boxes, everywhere. Even legacy applications like Notepad support the new Input Panel, without any effort of their own. Essentially, if you see a flashing caret (insertion point), expect to see the new Input Panel appear, when the pen hovers nearby.

In addition to maintaining backward compatibility for source code written to the 1.5 SDK, the Tablet PC team added a new purpose for the PenInputPanel class: you can use a PenInputPanel object to disable the new Input Panel. (How's that for irony?)

The new Input Panel is very cool, but every major change to applications' look and feel needs an opt-out mechanism. Undoubtedly, some folks have already created their own pop-up pen input panels, and they may feel the need to disable the new Input Panel. For more information and alternatives to disabling the new Input Panel, see Tracy Schultz's article, Disabling Tablet PC Input Panel.

Enhancing Recognition with Context Tagging and the New SetInputScope API

An ounce of prevention is worth a pound of cure. If this old adage applies anywhere, it certainly applies to the field of user interface designs for data entry—preventing a mistake is always preferable to relying on the user to correct it. The new Input Panel provides a wonderful correction experience, but it's at its best when correction is never needed at all.

New to Windows XP Tablet PC Edition 2005, Tablet PC applications have the ability to publish hints to the handwriting recognizer indicating what sort of data they're expecting in various fields. For example, the address bar in Internet Explorer has been modified to declare that it expects a URL or a file path. The updated Internet Explorer in Windows XP SP2 is effectively telling the Tablet PC's handwriting recognizer to bias recognition results toward strings that look like "https://www.", ".com", ".org", and the like. Dots and slashes are expected; spaces and other arcane punctuation marks are not allowed. These broad types of hints are known as input scopes, and it's flabbergasting to experience just how much they improve the usability of the Tablet PC.

Input scopes are similar in concept to the factoids found in previous versions of the Tablet PC SDK, but input scopes are much more powerful than factoids. With support for coercion, word lists, and phrase lists, an input scope is more like a fully formed RecognizerContext object than a factoid. In fact, multiple factoids can be combined together in the same input scope, along with word lists and even custom regular expressions, all to guide the recognizer. (If you've ever wondered why the Factoid property was designed as a string, instead of an enumeration, this is the reason why. The Tablet PC team had these advanced features planned, since day one.) The following example demonstrates the definition of a custom input scope, based on a regular expression and one of the built-in input scopes. This custom input scope will match any word beginning with the lowercase letter s, containing one or more characters, and ending with the lowercase letter p ("soup" or "stop", but not "instep" or "sign"):

theRecognizerContext.Factoid = "s(!IS_ONECHAR)+p";
theRecognizerContext.RecognitionFlags = RecognitionModes.Coerce;

But Internet Explorer is not an application built specifically for the Tablet PC platform. How does its address bar manage to take advantage of input scopes, in its recognition? After instantiating the address bar control, Internet Explorer makes a call to a simple new Win32 API function: SetInputScopes. This new function, defined in InputScope.h and implemented in MSCTF.dll, is a part of the Text Services Framework (TSF) API. The function merely maps an input scope to the given window handle. It's as simple as that.

It's so simple, in fact, that Microsoft has provided a declarative approach to performing the same enhancement. We can provide that same window-to-input-scope mapping by using a simple XML manifest file, without calling any new API functions. The new Tablet PC SDK includes a utility called the Context Tagging tool (CTagger.exe), which helps create and maintain these context manifest files (seen as .ctm files) for our favorite applications, without ever touching the source code. Here is a sample, short .ctm file produced by the Context Tagging Tool. The XML file shows the "numeric" input scope (IS_DIGITS) mapped to a text box in a simple Windows Forms application:

<?xml version="1.0" encoding="utf-8"?>
<ContextRoot xmlns="https://schemas.microsoft.com/2003/context-tagging" 
FileVersion="1.0.0.1">
 <ManifestHead ManifestVersion="1.0" 
xmlns="https://schemas.microsoft.com/2003/context-tagging">
  <Language Language_Name="Language-Neutral">
   <Application Application_Name="MyApp.exe">
    <Version Version_Number="1.0.1496.28397">
     <Manifest>
      <Screen>
       <AssociationList>
        <IScope>
         <Sign>
          <AccessibleName>Pick a number:</AccessibleName>
          <WindowClassName>WindowsForms10.EDIT.app2</WindowClassName>
          <WindowID>0</WindowID>
          <ControlName>Regex pattern:_WindowsForms10.EDIT.app2_0</ControlName>
          <ControlLocality>P[WindowsForms10.Window.8.app2,0]:PR[WindowsForms10.
BUTTON.app2,0]:NE[WindowsForms10.STATIC.app2,0]</ControlLocality>
          <AccessibleNameMandatory>true</AccessibleNameMandatory>
         </Sign>
         <Factoid>
          <InputScopeEnumList>
           <InputScopeEnum>IS_DIGITS</InputScopeEnum>
          </InputScopeEnumList>
          <RegExpName>
          </RegExpName>
          <PhraseListName>
          </PhraseListName>
         </Factoid>
        </IScope>
       </AssociationList>
      </Screen>
      <CustomDefinitionList />
      <WLFactoidList />
     </Manifest>
    </Version>
   </Application>
  </Language>
 </ManifestHead>
</ContextRoot>

The .ctm file is loaded at run-time, from alongside the application's .exe file. This convention is similar to the way .NET applications load configuration information from .config files, and the way Windows applications use .manfiest and .local files to control side-by-side versioning and application isolation, respectively.

Now, astute readers will note that it's not possible to map an input scope (or anything else for that matter) to an HWND, at design-time. Window handles are opaque object references, created at run-time. How can a design-time tool like CTagger.exe possibly hope to uniquely identify a window, without knowing its HWND? The truth is, it can't (always). But it usually can, by looking at the entire "path" to the child window of interest: from the window class name, accessibility name, and child identifier, to the text and class names of its sibling and parent windows, one can imagine constructing a reasonably unique identifier for a given text box at design-time. Windowless controls—as found in some Web browsers—and applications that dynamically generate their form layouts at run-time are the typical exceptions.

Support for Ink on the Web

Internet Explorer 5.5 and later includes support for hosting managed user controls (not ASP.NET web controls, mind you, but actual Windows Forms controls derived from System.Windows.Forms.UserControl) within HTML. Compared to the all-or-nothing security model of Microsoft ActiveX®, the security model of the .NET Framework allows for much, much finer-grained control over what these user controls are allowed to do and what they're not allowed to do. This security model—known as code access security—is based largely on Internet Explorer's concept of security zones:

Local Intranet zone
Trusted Site zone
Internet zone
Restricted Sites zone
Local Machine zone

For the uninitiated, here's a brief introduction to how code access security works: code is granted a set of permissions based on evidence. Most commonly, evidence takes the form of information about where the code came from (which zone, or what host) or who wrote it (public key hash, or Microsoft Authenticode® signature). For version 1.1 of the Microsoft .NET Framework, the policy mappings for the default code access security grant no permissions whatsoever to code from the Restricted Sites zone, full permissions to code from the Local Machine zone, and a reasonably conservative set of permissions to code from any of the zones in between. Good background articles on code access security and hosting managed .NET user controls in HTML can be found all over the Web, and in MSDN:

Using Libraries From Partially Trusted Code
Code Access Security in Practice
Host Secure, Lightweight Client-Side Controls in Microsoft Internet Explorer

Suffice to say, in previous versions of the Tablet PC Platform SDK, the Microsoft.Ink assembly was not declared safe for use in partially-trusted contexts, at all. This means that, by default, only code obtained from the My Computer zone—normal applications that you run from your local hard drive—could load Microsoft.Ink.dll. Windows XP Tablet PC Edition 2005 changes this. The security bar has been cleared, and the Microsoft.Ink assembly now bears the AllowPartiallyTrustedCallers attribute. This means we're now free to use Microsoft.Ink in partially trusted contexts—in Internet Explorer Web applications, for example, or in rich, full-fledged Windows Forms applications that are deployed on Internet Web servers by using the .NET Framework's No-Touch Deployment.

The set of permissions granted to Ink applications will vary, depending on the security context and the user's code access security configuration. It's this permission set that constrains which parts of the Tablet PC platform API we're actually allowed to call. For example, an application running in the Local Intranet zone will, by default, have full, unrestricted access to the user's clipboard. On the other hand, an application running from the Internet or Trusted Sites zones will have the ability to copy but not paste. Care must be taken to design applications which fail gracefully, when unable to perform certain operations due to code access security constraints.

Finally, many Web applications will need to support both Tablet PC and down-level (non-Tablet PC) clients. In previous versions of the Tablet PC platform, the HTTP User-Agent string was no different than on any other Windows XP system, so it was very difficult for Web applications to detect Tablet PC clients and respond with appropriate content. Windows XP Tablet PC Edition 2005 appends "Tablet PC 1.7" to the HTTP User-Agent string transmitted by Internet Explorer. Here's what the User-Agent string looks like on my RC2 build of Windows XP Tablet PC Edition 2005:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET 
CLR 1.1.4322; Tablet PC 1.7)

It's a small change that solves a very big problem.

The New RealTimeStylus API Layer

In previous versions of the Tablet PC platform, collection of Ink objects began and ended with the InkCollector or InkOverlay classes. Both are very high level abstractions of pen input, modeling Ink as a sequence of Stroke objects. This works very well for the vast majority of Tablet PC applications; nevertheless, developers have asked for tighter control over the Ink collection and rendering mechanisms. Microsoft has delivered, in the form of the new RealTimeStylus API. This new API appears in the same Microsoft.Ink assembly which we know and love, but in a new namespace: Microsoft.StylusInput.

One notable new capability of the RealTimeStylus API is the ability to filter or constrain the digitizer packet data as it comes in. The Tablet PC SDK 1.7 contains a sample application which does just that (developers of CAD and drawing applications take note). The RealTimeStylus API is like a tap into the digitizer data pipeline. The RealTimeStylus API works as a sequence of plug-ins, each of which has a turn at processing the packet data, analyzing it, filtering it, and modifying it before passing the data on to the next stage.

The RealTimeStylus API is not for every app, but for the class of applications that need it, it should prove invaluable.

SDK Versioning: Where Are We?

The Tablet PC may be getting a face lift, but the versioning issues around Microsoft.Ink may look a little confusing. Let's review: the initial release of the Tablet PC in November 2002 had Microsoft.Ink.dll stamped as 1.0.2201.0. Some months later, in 2003, version 1.5 of the SDK was released with the version for Microsoft.Ink adjusted to 1.0.2201.2. However, version 1.5 of the SDK also included a brand new assembly, Microsoft.Ink.15 (version 1.5.3023.0). This new "dot one-five" assembly contained the Divider and PenInputPanel classes.

Version 1.7 of the Tablet PC SDK brings good news and bad news to this versioning story. The bad news is that a few things have been moved around, renewing the complexity of versioning issues for Tablet PC application developers—but that's almost inevitable, especially given this release's unusual run-time dependence on an OS service pack. The good news is that Microsoft is heading in the right direction, making it simpler for the Tablet PC SDK to grow and evolve, in the future.

The version of Microsoft.Ink.dll in the RC2 release of Windows XP SP2 is 1.7.2600.2149 (That last number of the version will be different in the official release of the service pack.) This assembly—which is still built against the .NET Framework version 1.0 runtime—includes all of the classes from the original Microsoft.Ink assembly plus the Divider and PenInputPanel classes from the Microsoft.Ink.15 assembly, which is no longer part of the SDK.

Let me repeat that: The Microsoft.Ink.15 assembly is gone. In theory, this shouldn't break anything. The Microsoft.Ink assembly now contains all the classes it once exposed. Microsoft.Ink.15.dll no longer ships with the new SDK; however, you're still free to redistribute it with applications which depend on it. Indeed, you will likely continue to find it preinstalled in the global assembly cache, alongside builds 1.0.2201.0 and 1.0.2201.2 of Microsoft.Ink.dll, on most newer OEM images of Windows XP Tablet PC Edition.

However, in practice it's often hard to predict just when you might sit down, say, at a coworker's machine, and find that she's missing the Microsoft.Ink.15 runtime libraries you need to have her help test or debug your application. This might be a pretty common occurrence if your development team makes a slow transition to Lonestar. The installation of the Tablet PC SDK 1.7 will remove version 1.5 the SDK, including the version 1.5 assemblies from the global assembly cache, if they're not being referenced by any other installed applications. The usual symptom is a FileNotFound exception upon application startup, with a Fusion log dump showing the system failed to find Microsoft.Ink v1.0.2201.2.

Simply recompiling against the v1.7 libraries would fix the problem easily enough, but that's not always practical. (Of course, redistributing the v1.5 libraries via their merge modules would fix the problem too, but that's often even less practical, for ad-hoc testing and debugging sessions.) If you find yourself in this situation, unable or unwilling to rebuild against the new SDK, but needing to run a quick test on a Tablet PC without the 1.5 libraries installed, there are other options. One option is to provide an application configuration file which redirects requests for the older assemblies to the new version of Microsoft.Ink. Such a .config file might look something like this:

<?xml version="1.0" ?>
<configuration>
 <runtime>
  <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
   <dependentAssembly>
    <assemblyIdentity name="Microsoft.Ink" 
publicKeyToken="31bf3856ad364e35" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-1.5.65535.65535" 
newVersion="1.7.2600.2149" />
   </dependentAssembly>
   <dependentAssembly>
    <assemblyIdentity name="Microsoft.Ink.15" 
publicKeyToken="31bf3856ad364e35" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-1.5.65535.65535" 
newVersion="1.7.2600.2149" />
   </dependentAssembly>
  </assemblyBinding>
 </runtime>
</configuration>

The use of .config files to redirect assembly dependencies at deployment-time is a valuable skill to learn for troubleshooting unexpected versioning problems in the field. However, it's no substitute for the real thing. If your application depends on the version 1.5 Tablet PC runtime, you should install Microsoft.Ink v1.0.2201.2 on top of 1.7 by creating a setup based on tpcman12.msm, which can live side-by-side with Tpcman17.msm. For more information about versioning and Windows XP Tablet PC Edition 2005, see Using the Managed Library and Creating a Redistributable Setup in the Tablet PC SDK.

A few words about .NET vs. COM versioning are in order here. Versioning difficulties of the kind above, though rare, are ironic when you consider that Microsoft.Ink is a fairly thin managed wrapper around an unmanaged COM (Automation) API. The side-by-side versioning isolation provided by .NET is somewhat of an illusion. Under the hood, all three publicly released versions of the Microsoft.Ink assembly wrap the exact same COM class IDs. In other words, our .NET applications will load the latest installed unmanaged implementation at run-time, just as if we were all still COM developers. This can be an important detail to keep in mind when testing and debugging your applications on various editions of the Tablet PC Platform.

Conclusion

We've covered a lot of ground in this first column on Lonestar. The breadth of new features and improvements in Windows XP Tablet PC Edition 2005 is enormous. Add to that the immense list of changes in the rest of Windows XP Service Pack 2, and it's a lot to be excited about. Or perhaps it's a lot to be worried about, depending on your point of view. Windows XP Service Pack 2 will, almost certainly, break some existing applications. From the Windows Firewall and Data Execution Prevention (DEP) to Internet Explorer's new popup blocker and strict security around Microsoft ActiveX controls—all on by default—it's clear that the Windows team is emphasizing security over application compatibility. That is great news, in my opinion, even if it makes our jobs a little cumbersome in the short term.

The same can be said of Windows XP Tablet PC Edition 2005. It's great news, but it might make our jobs a little cumbersome in the short term. The road will be bumpiest over the next few months, as we ISV's attempt to support products built on a mix of Lonestar and pre-Lonestar Tablet PC, but the effort is justified. The improvements in the recognition experience, the full-fledged Web support, and the high performance RealTimeStylus API are just what the doctor ordered for our Tablet PC applications to really take flight.