How to interact with a background app in Cortana (XAML)

Article
08/31/2015

Learn how a user can interact with a background app through the Cortana voice and canvas during the execution of a voice command.

Voice commands with Cortana can include a rich user experience and interaction flow within Cortana that is controlled by the background app. The app can specify a number of different types of screens to support functionality that includes:

Successful completion
Hand-off
Progress
Confirmation
Disambiguation
Error

Prerequisites

This topic builds on Launch a background app with voice commands in Cortana. We continue here to demonstrate features with a trip planning and management app named Adventure Works.

To complete this tutorial, have a look through these topics to get familiar with the technologies discussed here.

Install Microsoft Visual Studio.
Get a developer license. For instructions, see Develop using Visual Studio 2013.
Create your first Windows Store app using C# or Visual Basic.
Roadmap for Windows Store apps using C# or Visual Basic
Learn about events with Events and routed events overview
See the VCD elements and attributes v1.2 reference for more info about VCD elements.
See Cortana design guidelines for how to integrate your app with Cortana and Speech design guidelines for helpful tips on designing a useful and engaging speech-enabled app.

Instructions

Completion screen

A completion screen provides the user with information about the completed voice command task.

Here we show how Cortana can display a list of travel itinerary results from the Adventure Works app for upcoming trips to Las Vegas.

Choose the feedback strings to be displayed and spoken by Cortana

Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.
Choose content tiles based on the action performed (optional)

Content tiles can provide additional context for the user and help keep the feedback strings concise.

Cortana supports the following content tile templates (only one template can be used on the completion screen):
- Title only
- Title with up to three lines of text
- Title with icon
- Title with icon and up to three lines of text
The icon can be:
- 68w x 68h
- 68w x 92h
- 280w x 140h
You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.

Show the successful completion screen

Here's an example of a successful completion screen with multiple content tiles.

var userMessage = new VoiceCommandUserMessage();
userMessage.DisplayMessage = "Here are your trips.";
userMessage.SpokenMessage = 
  "You have two trips to Vegas coming up.";

var destinationsContentTiles = new List<VoiceCommandContentTile>();

var destinationTile1 = new VoiceCommandContentTile();
destinationTile1.ContentTileType = 
  VoiceCommandContentTileType.TitleWith68x68IconAndText;
destinationTile1.AppLaunchArgument = “id_Vegas_001";
destinationTile1.Title = "Las Vegas Tech Conference";
destinationTile1.TextLine1 = "May 15th 2015";
destinationsContentTiles.Add(destinationTile1);

var destinationTile2 = new VoiceCommandContentTile();
destinationTile2.ContentTileType = 
  VoiceCommandContentTileType.TitleWith68x68IconAndText;
destinationTile2.AppLaunchArgument = “id_Vegas_002";
destinationTile2.Title = "Fun in Vegas";
destinationTile2.TextLine1 = "August 24th 2015";
destinationsContentTiles.Add(destinationTile2);

var response = 
  VoiceCommandResponse.CreateResponse(userMessage, destinationsContentTiles);

response.AppLaunchArgument = “destination=Las Vegas";

await voiceServiceConnection.ReportSuccessAsync(response);

Hand-off screen

Once a voice command is recognized, Cortana must present feedback in approximately .5 seconds. If the app service cannot complete the action specified by the voice command within .5 seconds, Cortana presents the user with a hand-off screen for up to 5 seconds.

Here's an example of a hand-off screen for the Adventure Works app. In this example, a user has queried Cortana for upcoming flights to Las Vegas. The hand-off screen includes a message customized with the app service name, an icon, and the Feedback string declared in the VCD file.

Progress screen

Once a voice command is recognized, Cortana must present feedback in approximately .5 seconds. If the app service requires more time to complete the action, it can provide a progress screen to inform the user that the voice command is being actively handled.

Cortana shows a progress screen for a maximum of 5 seconds. After 5 seconds, Cortana presents the user with an error message and the app service is closed. If the app service needs more than 5 seconds to complete the action, it can continue to update Cortana with progress screens.

Here's an example of a hand-off screen for the Adventure Works app. In this example, a user has canceled a trip to Las Vegas through Cortana. The progress screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

Choose the feedback strings to be displayed and spoken by Cortana

Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.
Choose content tiles based on the action performed (optional)

Content tiles can provide additional context for the user and help keep the feedback strings concise.

Cortana supports the following content tile templates (only one template can be used on the completion screen):
- Title only
- Title with up to three lines of text
- Title with icon
- Title with icon and up to three lines of text
The icon can be:
- 68w x 68h
- 68w x 92h
- 280w x 140h
You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.
Build the response

Call ReportProgressAsync to show the progress screen in Cortana.

Show the progress screen

Here's an example of a progress screen with a content tile.

var userMessage = new VoiceCommandUserMessage();

var destinationsContentTiles = new List<VoiceCommandContentTile>();

destinationsContentTiles.Add(selectedDestination);

var response = 
  VoiceCommandResponse.CreateResponse(userMessage, destinationsContentTiles);

response.AppLaunchArgument = "destination=Las Vegas";
await voiceServiceConnection.ReportProgressAsync(response);

Confirmation screen

When an action specified by a voice command is irreversible, has a significant impact, or the recognition confidence is not high, an app service can request confirmation.

Here's an example of a confirmation screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. The app service has provided Cortana with a confirmation screen that prompts the user for a yes or no answer before canceling the trip.

If the user says something other than "Yes" or "No", Cortana cannot determine the answer to the question. In this case, Cortana prompts the user with a similar question provided by the app service.

On the second prompt, if the user still doesn’t say "Yes" or "No", Cortana prompts the user a third time with the same question prefixed with an apology. If the user still doesn’t say "Yes" or "No", Cortana stops listening for voice input and asks the user to tap one of the buttons instead.

The confirmation screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

Choose the feedback strings to be displayed and spoken by Cortana

Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.
Choose content tiles based on the action performed (optional)

Content tiles can provide additional context for the user and help keep the feedback strings concise.

Cortana supports the following content tile templates (only one template can be used on the completion screen):
- Title only
- Title with up to three lines of text
- Title with icon
- Title with icon and up to three lines of text
The icon can be:
- 68w x 68h
- 68w x 92h
- 280w x 140h
You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.
Build the response

Call RequestConfirmationAsync to show the confirmation screen in Cortana.

Show the confirmation screen

Here's an example of a confirmation screen with a content tile.

var userPrompt = new VoiceCommandUserMessage();
userPrompt.DisplayMessage = userPrompt.SpokenMessage = 
  "Are you sure you want to cancel the trip to Las Vegas?”;

var userReprompt = new VoiceCommandUserMessage();
userReprompt.DisplayMessage = 
  userReprompt.SpokenMessage = "Do you want to cancel this trip to Las Vegas?"; 

userPrompt.DisplayMessage = “Cancel this trip?”;
userPrompt.SpokenMessage ="Do you wanna cancel this trip to Vegas?”;

var userReprompt = new VoiceCommandUserMessage();
userReprompt.DisplayMessage = “Did you want to cancel this trip?”;
userReprompt.SpokenMessage = "Did you wanna cancel this trip?"; 

var destinationsContentTiles = new List<VoiceCommandContentTile>();

var destinationTile = new VoiceCommandContentTile();
destinationTile.ContentTileType = 
  VoiceCommandContentTileType.TitleWith68x68IconAndText;
destinationTile.Title = "Vegas Tech Conference";

destinationTile.TextLine1 = "May 15th";


destinationsContentTiles.Add(destinationTile);

var response = 
  VoiceCommandResponse.CreateResponseForPrompt(
    userPrompt, userReprompt, destinationsContentTiles);

var voiceCommandConfirmation = 
  await voiceServiceConnection.RequestConfirmationAsync(response);

if (voiceCommandConfirmation != null)
{
   // Use the voiceCommandConfirmation.Confirmed to take action.
   // Call Cortana to present the next screen in .5 seconds 
   // and avoid a transition screen.
}

Disambiguation screen

When an action specified by a voice command has more than one possible outcome, an app service can request more info from the user.

Here's an example of a disambiguation screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. However, the user has two trips to Las Vegas on different dates and the app service cannot complete the action without the user selecting the intended trip.

The app service provides Cortana with a disambiguation screen that prompts the user to make a selection from a list of matching trips, before it cancels any.

In this case, Cortana prompts the user with a similar question provided by the app service.

On the second prompt, if the user still doesn’t say something that can be used to identify the selection, Cortana prompts the user a third time with the same question prefixed with an apology. If the user still doesn’t say something that can be used to identify the selection, Cortana stops listening for voice input and asks the user to tap one of the buttons instead.

The disambiguation screen includes a message customized for the action, an icon, and a content tile with information about the trip being canceled.

Choose the feedback strings to be displayed and spoken by Cortana

Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.
Choose content tiles based on the action performed (optional)

Content tiles can provide additional context for the user and help keep the feedback strings concise.

Cortana supports the following content tile templates (only one template can be used on the completion screen):
- Title only
- Title with up to three lines of text
- Title with icon
- Title with icon and up to three lines of text
The icon can be:
- 68w x 68h
- 68w x 92h
- 280w x 140h
You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.
Build the response

Call RequestDisambiguationAsync to show the disambiguation screen in Cortana.

Show the disambiguation screen

Here's an example of a disambiguation screen with content tiles.

// Create a VoiceCommandUserMessage for the initial question.  
var userPrompt = new VoiceCommandUserMessage();
userPrompt.DisplayMessage = "Which one do you want to cancel?";
userPrompt.SpokenMessage = 
  “Which Vegas trip do you wanna cancel? Vegas Tech Conference or Fun in Vegas?”;


// Create a VoiceCommandUserMessage for the second question,
// in case Cortana needs to reprompt. 
var userReprompt = new VoiceCommandUserMessage();
userReprompt.DisplayMessage = “Which one did you want to cancel?”;
userReprompt.SpokenMessage = "Which one did you wanna to cancel?";

// Create the list of content tiles to show the selection items.
var destinationsContentTiles = new List<VoiceCommandContentTile>();

var destinationTile = new VoiceCommandContentTile();
destinationTile.ContentTileType = 
  VoiceCommandContentTileType.TitleWith68x68IconAndText;

// The AppContext is optional. 
// Replace this value with something specific to your app. 
destinationTile.AppContext = "id_Vegas_001";
destinationTile.Title = "Vegas Tech Conference";

destinationTile.TextLine1 = "May 15th";


destinationsContentTiles.Add(destinationTile);

var destination2 = new VoiceCommandContentTile();
destination2.ContentTileType = 
  VoiceCommandContentTileType.TitleWith68x68IconAndText;
// The AppContext is optional. 
// Replace this value with something specific to your app. 
destination2.AppContext = "id_LasVegas_002";

destination2.Title = "Fun in Vegas";

destination2.TextLine1 = "August 24th";

destinationsContentTiles.Add(destination2);

// Create the disambiguation response.
var response = 
  VoiceCommandResponse.CreateResponseForPrompt(
    userPrompt, userReprompt, destinationsContentTiles);

// Request that Cortana shows the Disambiguation screen.
var voiceCommandDisambiguationResult = 
  await voiceServiceConnection.RequestDisambiguationAsync(response);

if (voiceCommandDisambiguationResult != null)
{
   // Use the voiceCommandDisambiguationResult.SelectedItem to take action.
   // Call Cortana to present the next screen in .5 seconds   
   // and avoid a transition screen. 
}

Error screen

When an action specified by a voice command cannot be completed, an app service can provide an error screen.

Here's an example of an error screen for the Adventure Works app. In this example, a user has instructed the app service to cancel a trip to Las Vegas through Cortana. However, the user does not have any trips scheduled to Las Vegas.

The app service provides Cortana with an error screen that includes a message customized for the action, an icon, and the specific error message.

Choose the feedback strings to be displayed and spoken by Cortana

Follow the Cortana design guidelines for recommendations on composing strings that Cortana shows and speaks.
Choose content tiles based on the action performed (optional)

Content tiles can provide additional context for the user and help keep the feedback strings concise.

Cortana supports the following content tile templates (only one template can be used on the completion screen):
- Title only
- Title with up to three lines of text
- Title with icon
- Title with icon and up to three lines of text
The icon can be:
- 68w x 68h
- 68w x 92h
- 280w x 140h
You can also let users launch your app in the foreground by either tapping a tile or the text link to your app.
Build the response

Call ReportFailureAsync to show the error screen in Cortana.

Show the error screen

Here's an example of an error screen.

var userMessage = new VoiceCommandUserMessage();
userMessage.DisplayMessage = userMessage.SpokenMessage = 
  "Sorry, you don't have any trips to Las Vegas";

var response = VoiceCommandResponse.CreateResponse(userMessage);

response.AppLaunchArgument = "showUpcomingTrips";
await voiceServiceConnection.ReportFailureAsync(response);