Handwriting Recognition Improvements in Microsoft Windows XP Tablet PC Edition Service Pack 2
Li Lu-Porter and Antje Helfrich
Tablet PC team
Microsoft® Windows® XP Tablet PC Edition Service Pack 2 (SP2)
Summary: Learn about the latest improvements to handwriting recognition in Windows XP Tablet PC Edition Service Pack 2 (SP2). This paper provides an overview of the improvements to the handwriting recognizers for western languages (English, French, and German) and East Asian languages (Japanese, Korean, Simplified Chinese, and Traditional Chinese). (6 printed pages)
Windows XP Tablet PC Edition SP2 provides a compelling end-to-end text input experience on a tablet computer and improves the overall handwriting recognition accuracy for key user tasks, such as replying to or composing an e-mail message, entering an e-mail address, editing a document, writing notes to self, and creating a to-do list.
To define the handwriting recognition features and capabilities in Windows XP Tablet PC Edition SP2, the Tablet PC team sought subjective and objective user feedback. Handwriting recognition is a complex area. Individual differences and situational factors influence the user experience and overall user satisfaction. Influential factors include:
- The recognition engine's accuracy and performance.
- The user interface design.
- The user's handwriting style, such as print, cursive, or block letters.
- User tasks, such as writing quick notes, replying to an important e-mail message, or carefully editing a document.
- The type of content that the user enters, such as text, acronyms, proper nouns, numbers, and Web addresses.
- The user's handedness, that is, whether the user is right-handed or left-handed.
- Application settings, specifically, how the user's application calls the recognizer.
Recognizer accuracy—defined as a quantitative measure of error—is insufficient for understanding the end-to-end user experience. Though accuracy affects the overall usability of the Windows XP Tablet PC operating system, other factors also contribute to user satisfaction, such as effectiveness and efficiency.
Measuring user satisfaction in handwriting recognition is a challenging, somewhat experimental task. The Tablet PC team launched a field study that focuses on handwriting recognition of users in the United States and a key international market, Germany. The study uses several methodologies, including interviews, surveys, and observations to better understand usage behavior, pain points in the system, and users' overall satisfaction with handwriting recognition.
The Tablet PC team also conducted ongoing user trials of Windows XP Tablet PC Edition SP2. We also compared the recognizers in Windows XP Tablet PC Edition with the recognizers in Windows XP Tablet PC Edition SP2 to ensure quality and performance.
The research resulted in several improvements to the western language recognizers. One of the most important improvements pertains to the contents of a text control.
Use of Context
English, French, and German use descriptive information about the contents of a text control, known as context, to improve recognition accuracy in Windows XP Tablet PC Edition SP2. Context is application-specific information about a control that tells the recognizer what type of user input to expect. A particular type of user input is called an input scope. Examples of predefined input scopes are e-mail addresses, URLs, personal names, postal addresses, and telephone numbers. Though Windows XP Tablet PC Edition SP2 supports factoids from Windows XP Tablet PC Edition, input scopes are recommended for development for the western recognizers. Developers can also set a custom input scope by using a regular expression.
You can set the input scope for a control in one of three ways:
For ink-enabled controls:
- You can set the Factoid property of the RecognizerContext object and pass in an input scope value.
For controls that are not ink-enabled:
- You can use the SetInputScope application programming interface (API). This method requires that you modify your application's code.
- You can use the Context Tagging Tool, which is a utility that developers use to tag an existing application with the context information. The tool is distributed with the Tablet PC Edition Development Kit 1.7. Because this method records context information in an XML file that is separate from the application files, it is not necessary to modify existing ink-unaware applications.
Context has been enabled in Windows XP Tablet PC Edition SP2 for the most commonly used data fields in six Microsoft Office 2003 applications. These applications are Microsoft Word, Excel, Outlook, PowerPoint, InfoPath, and OneNote. To take advantage of context in Office applications, users must install the Office 2003 Context Tagging Manifests, which will be offered as a Web download. When using these applications, users will experience much better handwriting recognition in fields whose syntax does not conform to the rules of normal written language. For example, users who write e-mail addresses in the To field in Outlook experience significantly fewer errors.
Context has also been enabled for the Address bar in Microsoft Internet Explorer, which has improved recognition when users write URLs or file paths. Related improvements that affect these scenarios are a URL-specific key pad in Tablet PC Input Panel and better recognition for common character sequences in URLs and file paths, such as "http://," "com," and forward slashes.
For more information about using context, see Windows XP Tablet PC Edition.
Other Recognition Improvements
Windows XP Tablet PC Edition SP2 includes several other key recognition improvements for western recognizers.
- In-word context: Leveraging the left and right context along with the language model and lexicon help Input Panel generate the right results. The inclusion of such context is especially helpful when users are correcting their handwritten text. For example, if Input Panel recognizes the word "dictionary" incorrectly as "dictum," and the user chooses to correct "um," the recognizer can use the context of "dict" to find the ending "ionary."
- Extended locale support for western recognizers: Recognizers are made available for all locales in which the respective language is spoken. For example, the English recognizer is enabled for Locale English (Australia), and the German recognizer is enabled for Locale German (Austria).
- Several specific improvements to increase recognition accuracy, as described in the following table.
|Improved recognition rate||Handwriting recognition has improved for short prose, making it faster and easier to reply to or compose e-mail messages, annotate documents, or write notes.
Special heuristics were implemented to guide the recognition of ambiguous character shapes. For example, a round shape is recognized as the letter "O" rather than the digit "0," depending on the width of the character.
|Improved recognition for single characters||Improved recognition of single characters especially benefits the user when correcting individual characters. For example, distinctions between lowercase and uppercase characters results in a higher accuracy rate, including pairs of characters with similar shapes in lowercase and uppercase, such as sS and wW.
The recognition rate improved for 94 out of 98 U.S. English characters, including punctuation, such as the comma and the exclamation mark, and special symbols, such as the @ symbol or the backslash \.
|Language lexicon improvements||Common abbreviations, colloquialisms, neologisms, greetings, IT, and technical terms were added to the lexicon.
Rare words that the recognizer may confuse with more common words have been removed. For example:
If needed, users can add rare forms or inflections to the lexicon.
|Improved recognition of delayed strokes||The user can add quotes or similar strokes after a word is written, which makes using handwriting more flexible and natural. It also improves the recognition of contractions, such as "I'll" and "would've."|
In addition, the following improvements were made for specific western languages.
|Better recognition of the English |
|The character "I" is now less likely to be returned as a digit 1 or lowercase letter l when it occurs in prose.|
|Better recognition of characters with diacritics and ligatures||Examples include "é" and "æ."|
|Conforms to French spelling reform||The addition of 20,000 new spellings (including inflections) to the handwriting lexicon accommodates "La réforme de l'orthographe." New and existing spellings are recognized, such as "connaitre" (new) and "connaître" (old), "couter" (new) and "coûter" (old).|
|Better recognition of Umlaut characters and "ß" character.||Examples include the letters "ä" and "ß."|
|Improved recognition rate for compound nouns||A dynamic compounding algorithm was implemented, so that words such as "Elektronikunternehmen," "Konkurrenzmarkt," "Wahlkampfhilfe" are now recognized more reliably.|
|More post-reform spellings added||Many words that were formerly spelled with "ß" that are now spelled with "ss" are included, such as Abendimbiss and Stress.|
Windows XP Tablet PC Edition SP2 includes the following improvements for recognizers of East Asian characters. East Asian languages include Japanese, Korean, Simplified Chinese, and Traditional Chinese.
- LinedInput and FreeInput mode support: All recognizers of East Asian characters now support these two recognizer capabilities. Lined mode writing occurs in Input Panel and WordPad. Free mode writing occurs in OneNote and Windows Journal. The accuracy of LinedInput and FreeInput has improved significantly due to training and fine-tuning of the recognizers' internal neural network with a larger number of handwriting samples; leveraging sophisticated language model; and improving lexicon support.
- Cursive support: All East Asian recognizers now support cursive writing. Most East Asian users write cursively in one form or another, especially for quick informal tasks, such as note taking, composing a short e-mail message, and chatting. Better cursive support means a more efficient handwriting-to-text conversion process.
- Recognition improvements for small and large Katakana and Hiragana characters, and lowercase and uppercase characters: Special heuristics and better use of user interface height information enables the recognizer to better distinguish these characters.
- Characters frequently used for Japanese names: These characters have been added to the commonly used characters set to improve recognition.
- Improved recognition accuracy: Special attention has been given to factoid data, such as alphanumeric characters, telephone numbers, and dates. Improvements range from noticeable to significantly better. Work is underway to further improve the accuracy of less structured factoid data, such as e-mail addresses and Web addresses. In addition, adjusted character frequencies have tuned recognition results for characters that are easily confused with ones with similar shapes.
Windows XP Tablet PC Edition SP2 improves the overall accuracy of each shipping recognizer. For western languages, the key improvements include the first version of context, better core recognition, and targeted improvements for character and word recognition during correction. For East Asian languages, significant accuracy improvements and the support of cursive writing help give users the handwriting capabilities that they want.