Broadcasting IM Text Based on Speech Recognition in a UCMA Application: Code Listing (Part 3 of 3)
Summary: Combine speech recognition, Microsoft Lync 2010 Automation API, and Microsoft Unified Communications Managed API (UCMA) 3.0 to broadcast urgent text to clients and customers.
Applies to: Microsoft Unified Communications Managed API (UCMA) 3.0 Core SDK | Microsoft Lync 2010 SDK | Microsoft Speech Platform SDK
Published: November 2011 | Provided by: John Clarkson, Microsoft | About the Author
Contents
Conclusion
Additional Resources
This article is the last in a three-part series of articles about how to use speech recognition to broadcast instant messaging text from a Microsoft Unified Communications Managed API (UCMA) 3.0 application.
Grammar File
The following code example contains the Speech Recognition Grammar Specification (SRGS) XML grammar that is used for speech recognition in the UCMA 3.0 application. For more information about this kind of grammar, see Speech Recognition Grammar Specification Version 1.0, Semantic Interpretation for Speech Recognition (SISR) Version 1.0, and Speech Recognition.
<?xml version="1.0" encoding="UTF-8" ?>
<grammar version="1.0" xml:lang="en-US" mode="voice" root= "Main"
xmlns="http://www.w3.org/2001/06/grammar" tag-format="semantics/1.0">
<rule id="Main" scope="public">
<example>Buy Contoso maximum ten send</example>
<one-of>
<item>buy</item>
<item>sell</item>
</one-of>
<one-of>
<item>Fabrikam</item>
<item>Contoso</item>
</one-of>
<one-of>
<item>maximum</item>
<item>minimum</item>
</one-of>
<one-of>
<item>one</item>
<item>ten</item>
<item>twenty</item>
</one-of>
<one-of>
<item>send</item>
</one-of>
</rule>
</grammar>
Program.cs
The following example contains the code from the Program.cs file in the UCMA project.
using System;
using System.Threading;
using Microsoft.Rtc.Collaboration;
using Microsoft.Rtc.Collaboration.AudioVideo;
using Microsoft.Rtc.Signaling;
using Microsoft.Speech.Recognition;
using Microsoft.Speech.AudioFormat;
using Microsoft.Rtc.Collaboration.Sample.Common;
using Microsoft.Lync.Model;
using Microsoft.Lync.Model.Extensibility;
using System.Configuration;
namespace Microsoft.Rtc.Collaboration.Sample.IM_And_SpeechReco
{
public class UCMA_SR_IM
{
private UCMASampleHelper _helper;
private UserEndpoint _userEndpoint;
private AudioVideoCall _audioVideoCall;
private AudioVideoFlow _audioVideoFlow;
private string imMessageContent;
int index = 0;
ConversationWindow cWindow;
Automation automation;
// Wait handles are used to keep the main and worker threads synchronized.
private AutoResetEvent _waitForCallToBeAccepted = new AutoResetEvent(false);
private AutoResetEvent _waitForConnectorToStop = new AutoResetEvent(false);
private AutoResetEvent _waitForConversationToBeTerminated = new AutoResetEvent(false);
private AutoResetEvent _waitForShutdownEventCompleted = new AutoResetEvent(false);
private AutoResetEvent _waitForRecoCompleted = new AutoResetEvent(false);
private AutoResetEvent _waitForSendMethodCompleted = new AutoResetEvent(false);
static void Main(string[] args)
{
UCMA_SR_IM recommend = new UCMA_SR_IM();
recommend.Run();
}
public void Run()
{
// A helper class to take care of platform and endpoint setup and cleanup.
_helper = new UCMASampleHelper();
// Create a user endpoint using the network credential object.
_userEndpoint = _helper.CreateEstablishedUserEndpoint("Broadcast User");
// Register a delegate to be called when an incoming audio-video call arrives.
_userEndpoint.RegisterForIncomingCall<AudioVideoCall>(AudioVideoCall_Received);
// Wait for the incoming call to be accepted.
Console.WriteLine("Waiting for incoming call...");
_waitForCallToBeAccepted.WaitOne();
// Create a speech recognition connector and attach an AudioVideoFlow to it.
SpeechRecognitionConnector speechRecognitionConnector = new SpeechRecognitionConnector();
speechRecognitionConnector.AttachFlow(_audioVideoFlow);
// Start the speech recognition connector.
SpeechRecognitionStream stream = speechRecognitionConnector.Start();
// Create a speech recognition engine.
SpeechRecognitionEngine speechRecognitionEngine = new SpeechRecognitionEngine();
speechRecognitionEngine.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(SpeechRecognitionEngine_SpeechRecognized);
//Add a grammar.
Grammar gr = new Grammar(@"C:\Data\CodeSample\SRandIM\CodeReview2\CodeReview2\Buy.grxml", "Main");
speechRecognitionEngine.LoadGrammarAsync(gr);
//Attach audio stream to the SR engine.
SpeechAudioFormatInfo speechAudioFormatInfo = new SpeechAudioFormatInfo(8000, AudioBitsPerSample.Sixteen, Microsoft.Speech.AudioFormat.AudioChannel.Mono);
speechRecognitionEngine.SetInputToAudioStream(stream, speechAudioFormatInfo);
Console.WriteLine("\r\nGrammar loaded, say send to send IM.");
//Prepare the SR engine to perform multiple asynchronous recognitions.
speechRecognitionEngine.RecognizeAsync(RecognizeMode.Multiple);
//Pause the main thread until recognition completes.
_waitForConnectorToStop.WaitOne();
speechRecognitionConnector.Stop();
Console.WriteLine("connector stopped");
// Detach the flow from the speech recognition connector, to prevent the flow from being kept in memory.
speechRecognitionConnector.DetachFlow();
// Terminate the call, the conversation, and then unregister the
// endpoint from receiving an incoming call.
_audioVideoCall.BeginTerminate(CallTerminateCB, _audioVideoCall);
_waitForConversationToBeTerminated.WaitOne();
// Shut down the platform.
_helper.ShutdownPlatform();
}
public void SendIM()
{
Console.WriteLine("exiting");
//Get a Lync client automation object.
LyncClient client = LyncClient.GetClient();
automation = LyncClient.GetAutomation();
//Add two URIs to the list of IM addresses.
System.Collections.Generic.List<string> inviteeList = new System.Collections.Generic.List<string>();
inviteeList.Add(ConfigurationManager.AppSettings["UserURI"]);
inviteeList.Add(ConfigurationManager.AppSettings["UserURI2"]);
//Specify IM settings.
System.Collections.Generic.Dictionary<AutomationModalitySettings, object> mSettings = new System.Collections.Generic.Dictionary<AutomationModalitySettings, object>();
string messageText = ImMessageText();
mSettings.Add(AutomationModalitySettings.FirstInstantMessage, messageText);
mSettings.Add(AutomationModalitySettings.SendFirstInstantMessageImmediately, true);
//Broadcast the IM messages.
IAsyncResult ar = automation.BeginStartConversation(AutomationModalities.InstantMessage, inviteeList, mSettings, null, null);
cWindow = automation.EndStartConversation(ar);
}
//Build the IM message string.
string ImMessageText()
{
//Parse the recognition return
char[] delimiterCharacters = {' '};
string[] words = imMessageContent.Split(delimiterCharacters);
//Grab array elements
string msgTXT = "Hi this is your broker Kate Berger with an urgent recommendation to ";
string operation = words[0];
string name = words[1];
string minORmax = words[2];
string price = words[3];
if (operation == "buy")
{
msgTXT = msgTXT + "BUY " + name + " with a " + minORmax + " value of " + price + ".";
}
else
{
msgTXT = msgTXT + "SELL " + name + "with a " + minORmax + " value of " + price + ".";
}
return msgTXT;
}
#region EVENT HANDLERS
// Delegate that is called when an incoming AudioVideoCall arrives.
void AudioVideoCall_Received(object sender, CallReceivedEventArgs<AudioVideoCall> e)
{
_audioVideoCall = e.Call;
_audioVideoCall.AudioVideoFlowConfigurationRequested += this.AudioVideoCall_FlowConfigurationRequested;
// For logging purposes, register for notification of the StateChanged event on the call.
_audioVideoCall.StateChanged +=
new EventHandler<CallStateChangedEventArgs>(AudioVideoCall_StateChanged);
// Remote Participant URI represents the far end (caller) in this conversation.
Console.WriteLine("Call received from: " + e.RemoteParticipant.Uri);
// Now, accept the call. CallAcceptCB will run on the same thread.
_audioVideoCall.BeginAccept(CallAcceptCB, _audioVideoCall);
}
// Handles the StateChanged event on the incoming audio-video call.
void AudioVideoCall_StateChanged(object sender, CallStateChangedEventArgs e)
{
Console.WriteLine("Previous call state: " + e.PreviousState + "\nCurrent state: " + e.State);
}
// Handles the StateChanged event on the audio-video flow.
private void AudioVideoFlow_StateChanged(object sender, MediaFlowStateChangedEventArgs e)
{
// When the flow is active, media operations can begin.
if (e.State == MediaFlowState.Terminated)
{
// Detach the speech recognition connector, because the state of the flow is now Terminated.
AudioVideoFlow avFlow = (AudioVideoFlow)sender;
if (avFlow.SpeechRecognitionConnector != null)
{
avFlow.SpeechRecognitionConnector.DetachFlow();
}
}
}
public void AudioVideoCall_FlowConfigurationRequested(object sender, AudioVideoFlowConfigurationRequestedEventArgs e)
{
Console.WriteLine("Flow Created.");
_audioVideoFlow = e.Flow;
// Now that the flow is non-null, bind a handler for the StateChanged event.
// When the flow goes active, (as indicated by the StateChanged event) the application can take media-related actions on the flow.
_audioVideoFlow.StateChanged += new EventHandler<MediaFlowStateChangedEventArgs>(AudioVideoFlow_StateChanged);
}
void SpeechRecognitionEngine_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)
{
RecognitionResult result = e.Result;
if (result != null)
{
Console.WriteLine("Speech recognized: " + result.Text);
imMessageContent = result.Text;
if (result.Text.Contains("send"))
{
SendIM(); //Call the method to send IM messages.
_waitForSendMethodCompleted.WaitOne();
_waitForConnectorToStop.Set();
}
}
}
#endregion
#region CALLBACK METHODS
private void CallAcceptCB(IAsyncResult ar)
{
AudioVideoCall audioVideoCall = ar.AsyncState as AudioVideoCall;
try
{
// Determine whether the call was accepted successfully.
audioVideoCall.EndAccept(ar);
}
catch (RealTimeException exception)
{
// RealTimeException may be thrown on media or link-layer failures.
// A production application should catch additional exceptions, such as OperationTimeoutException,
// OperationTimeoutException, and CallOperationTimeoutException.
Console.WriteLine(exception.ToString());
}
finally
{
// Synchronize with main thread.
_waitForCallToBeAccepted.Set();
}
}
private void CallTerminateCB(IAsyncResult ar)
{
AudioVideoCall audioVideoCall = ar.AsyncState as AudioVideoCall;
// Finish terminating the incoming call.
audioVideoCall.EndTerminate(ar);
// Unregister this event handler now that the call has been terminated.
_audioVideoCall.StateChanged -= AudioVideoCall_StateChanged;
// Terminate the conversation.
_audioVideoCall.Conversation.BeginTerminate(ConversationTerminateCB, _audioVideoCall.Conversation);
}
private void ConversationTerminateCB(IAsyncResult ar)
{
Conversation conversation = ar.AsyncState as Conversation;
// Finish terminating the conversation.
conversation.EndTerminate(ar);
// Unregister for incoming calls.
_userEndpoint.UnregisterForIncomingCall<AudioVideoCall>(AudioVideoCall_Received);
// Synchronize with main thread.
_waitForConversationToBeTerminated.Set();
}
#endregion
}
}
Helper Class
The code appearing in this section is an abbreviated version of the code that appears in the UCMA 3.0 SDK UCMASampleHelper.cs file. The following code includes declarations and the following methods. To see the complete contents, in the folder that contains the UCMA 3.0 SDK, browse to the path SDK\Core\Sample Applications\QuckStarts\Common.
CreateEstablishedUserEndpoint
EstablishUserEndpoint
ReadUserSettings
CreateUserEndpoint
PromptUser
using System;
using System.Configuration;
using System.Globalization;
using System.Runtime.InteropServices;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading;
using Microsoft.Rtc.Collaboration;
using Microsoft.Rtc.Signaling;
namespace Microsoft.Rtc.Collaboration.Sample.Common
{
class UCMASampleHelper
{
private static ManualResetEvent _sampleFinished = new ManualResetEvent(false);
// The name of this application, to be used as the outgoing user agent string.
// The user agent string is put in outgoing message headers to indicate the Application used.
private static string _applicationName = "UCMASampleCode";
const string _sipPrefix = "sip:";
// These strings will be used as keys into the App.Config file to get information to avoid prompting. For most of these,
// Suffixes 1-N will be used on each subsequent call. eg. UserName1 will be used for the first user, UserName2 for the second, etc.
private static String _serverFQDNPrompt = "ServerFQDN";
private static String _userNamePrompt = "UserName";
private static String _userDomainPrompt = "UserDomain";
private static String _userURIPrompt = "UserURI";
// Construct the network credential that the UserEndpoint will use to authenticate to the Microsoft Lync Server.
private string _userName; // User name and password pair of a user enabled for Microsoft Lync Server.
private string _userPassword;
private string _userDomain; // Domain that this user is logging into. Note: This is the AD domain, not the portion of the SIP URI following the at sign.
private System.Net.NetworkCredential _credential;
// The URI and connection server of the user used.
private string _userURI; // This should be the URI of the user given above.
// The Server FQDN used.
private static string _serverFqdn;// The Microsoft Lync Server to log in to.
// Transport type used to communicate with your Microsoft Lync Server instance.
private Microsoft.Rtc.Signaling.SipTransportType _transportType = Microsoft.Rtc.Signaling.SipTransportType.Tls;
private static CollaborationPlatform _collabPlatform;
private static bool _isPlatformStarted;
private static CollaborationPlatform _serverCollabPlatform;
private AutoResetEvent _platformStartupCompleted = new AutoResetEvent(false);
private AutoResetEvent _endpointInitCompletedEvent = new AutoResetEvent(false);
private AutoResetEvent _platformShutdownCompletedEvent = new AutoResetEvent(false);
private UserEndpoint _userEndpoint;
private ApplicationEndpoint _applicationEndpoint;
// The FQDN of the machine that the application contact is targeted at.
private string _applicationHostFQDN;
// The port that the application contact is configured to receive data on.
private int _applicationPort = -1;
// The GRUU assigned to the application contact.
private string _applicationGruu;
// The URI of the contact being used.
private string _applicationContactURI;
private string _certificateFriendlyName;
private bool _useSuppliedCredentials;
private static int _appContactCount;
private static int _userCount = 1;
// Method to read user settings from app.config file or from the console prompts
// This method returns a UserEndpointSettings object. If you do not want to monitor LocalOwnerPresence, you may
// want to call the CreateEstablishedUserEndpoint method directly. Otherwise, you may call ReadUserSettings
// followed by CreateUserEndpoint, followed by EstablishUserEndpoint methods.
public UserEndpointSettings ReadUserSettings(string userFriendlyName)
{
UserEndpointSettings userEndpointSettings = null;
string prompt = string.Empty;
if (string.IsNullOrEmpty(userFriendlyName))
{
userFriendlyName = "Default User";
}
try
{
Console.WriteLine(string.Empty);
Console.WriteLine("Creating User Endpoint for {0}...", userFriendlyName);
Console.WriteLine();
if (ConfigurationManager.AppSettings[_serverFQDNPrompt + _userCount] != null)
{
_serverFqdn = ConfigurationManager.AppSettings[_serverFQDNPrompt + _userCount];
Console.WriteLine("Using {0} as Microsoft Lync Server", _serverFqdn);
}
else
{
// Prompt user for server FQDN. If server FQDN was entered before, then let the user use the saved value.
string localServer;
StringBuilder promptBuilder = new StringBuilder();
if (!string.IsNullOrEmpty(_serverFqdn))
{
promptBuilder.Append("Current Microsoft Lync Server = ");
promptBuilder.Append(_serverFqdn);
promptBuilder.AppendLine(". Please hit ENTER to retain this setting - OR - ");
}
promptBuilder.Append("Please enter the FQDN of the Microsoft Lync Server that the ");
promptBuilder.Append(userFriendlyName);
promptBuilder.Append(" endpoint is homed on => ");
localServer = PromptUser(promptBuilder.ToString(), null);
if (!String.IsNullOrEmpty(localServer))
{
_serverFqdn = localServer;
}
}
// Prompt user for user name
prompt = String.Concat("Please enter the User Name for ",
userFriendlyName,
" (or hit the ENTER key to use current credentials)\r\n" +
"Please enter the User Name => ");
_userName = PromptUser(prompt, _userNamePrompt + _userCount);
// If user name is empty, use current credentials
if (string.IsNullOrEmpty(_userName))
{
Console.WriteLine("Username was empty - using current credentials...");
_useSuppliedCredentials = true;
}
else
{
// Prompt for password
prompt = String.Concat("Enter the User Password for ", userFriendlyName, " => ");
_userPassword = PromptUser(prompt, null);
prompt = String.Concat("Please enter the User Domain for ", userFriendlyName, " => ");
_userDomain = PromptUser(prompt, _userDomainPrompt + _userCount);
}
// Prompt user for user URI
prompt = String.Concat("Please enter the User URI for ", userFriendlyName, " in the User@Host format => ");
_userURI = PromptUser(prompt, _userURIPrompt + _userCount);
if (!(_userURI.ToLower().StartsWith("sip:") || _userURI.ToLower().StartsWith("tel:")))
_userURI = "sip:" + _userURI;
// Increment the last user number
_userCount++;
// Initalize and register the endpoint, using the credentials of the user the application will be acting as.
// NOTE: the _userURI should always be of the form "sip:user@host"
userEndpointSettings = new UserEndpointSettings(_userURI, _serverFqdn);
if (!_useSuppliedCredentials)
{
_credential = new System.Net.NetworkCredential(_userName, _userPassword, _userDomain);
userEndpointSettings.Credential = _credential;
}
else
{
userEndpointSettings.Credential = System.Net.CredentialCache.DefaultNetworkCredentials;
}
}
catch (InvalidOperationException iOpEx)
{
// Invalid Operation Exception should only be thrown on poorly-entered input.
Console.WriteLine("Invalid Operation Exception: " + iOpEx.ToString());
}
return userEndpointSettings;
}
// Method to create an endpoint given a UserEndpointSettings object.
// This method returns a UserEndpoint object so that you can wire up Endpoint-specific event handlers.
// If you do not want to get endpoint specific event information at the time the endpoint is established, you may
// want to call the CreateEstablishedUserEndpoint method directly. Otherwise, you may call ReadUserSettings
// followed by CreateUserEndpoint, followed by EstablishUserEndpoint methods.
public UserEndpoint CreateUserEndpoint(UserEndpointSettings userEndpointSettings)
{
// Reuse platform instance so that all endpoints share the same platform.
if (_collabPlatform == null)
{
// Initalize and startup the platform.
ClientPlatformSettings clientPlatformSettings = new ClientPlatformSettings(_applicationName, _transportType);
_collabPlatform = new CollaborationPlatform(clientPlatformSettings);
}
_userEndpoint = new UserEndpoint(_collabPlatform, userEndpointSettings);
return _userEndpoint;
}
// Method to establish an already created UserEndpoint.
// This method returns an established UserEndpoint object. If you do not want to monitor LocalOwnerPresence, you may
// want to call the CreateEstablishedUserEndpoint method directly. Otherwise, you may call ReadUserSettings
// followed by CreateUserEndpoint, followed by EstablishUserEndpoint methods.
public bool EstablishUserEndpoint(UserEndpoint userEndpoint)
{
// Startup the platform, if not already
if (_isPlatformStarted == false)
{
userEndpoint.Platform.BeginStartup(EndPlatformStartup, userEndpoint.Platform);
// Sync; wait for the platform startup to complete.
_platformStartupCompleted.WaitOne();
Console.WriteLine("Platform started...");
_isPlatformStarted = true;
}
// Establish the user endpoint
userEndpoint.BeginEstablish(EndEndpointEstablish, userEndpoint);
// Sync; wait for the registration to complete.
_endpointInitCompletedEvent.WaitOne();
Console.WriteLine("Endpoint established...");
return true;
}
// Method to create an established UserEndpoint.
// This method returns an established UserEndpoint object. If you do not want to monitor LocalOwnerPresence, you may
// want to call this CreateEstablishedUserEndpoint method directly. Otherwise, you may call ReadUserSettings
// followed by CreateUserEndpoint, followed by EstablishUserEndpoint methods.
public UserEndpoint CreateEstablishedUserEndpoint(string endpointFriendlyName)
{
UserEndpointSettings userEndpointSettings;
UserEndpoint userEndpoint = null;
try
{
// Read user settings
userEndpointSettings = ReadUserSettings(endpointFriendlyName);
// Create User Endpoint
userEndpoint = CreateUserEndpoint(userEndpointSettings);
// Establish the user endpoint
EstablishUserEndpoint(userEndpoint);
}
catch (InvalidOperationException iOpEx)
{
// Invalid Operation Exception should only be thrown on poorly-entered input.
Console.WriteLine("Invalid Operation Exception: " + iOpEx.ToString());
}
return userEndpoint;
}
/// <summary>
/// If the 'key' is not found in app config, prompts the user using prompt text.
/// </summary>
/// <param name="promptText">If key is not found in app.Config, the user will be prompted for input using this parameter.</param>
/// <param name="key">Searches for this key in app.Config and returns if found. Pass null to always prompt.</param>
/// <returns>String value either from App.Config or user input.</returns>
public static string PromptUser(string promptText, string key)
{
String value;
if (String.IsNullOrEmpty(key) || ConfigurationManager.AppSettings[key] == null)
{
Console.WriteLine(string.Empty);
Console.Write(promptText);
value = Console.ReadLine();
}
else
{
value = ConfigurationManager.AppSettings[key];
Console.WriteLine("Using keypair {0} - {1} from AppSettings...", key, value);
}
return value;
}
}
}
Conclusion
This article describes how to perform speech recognition in a UCMA 3.0 application, use the recognition result to compose message text, and then use the Lync 2010 Automation API to broadcast that text to colleagues and customers in an instant message.
Additional Resources
For more information, see the following resources:
About the Author
John Clarkson is a programming writer with the Microsoft Lync product team.