Webcam and Device Overview
Silverlight 4 enables the ability to capture source input from audio and video devices such as webcams and microphones. This overview covers connecting and interacting with audio and video devices in your Silverlight applications.
This topic contains the following sections.
In Silverlight 4 you can capture and display audio and video from media devices, such as webcams, TV tuner, and microphones. This enables a number of scenarios which can enhance the user experience of your Silverlight applications. Some of these scenarios are capturing and displaying images, uploading profile pictures to social networking applications, video mail and video diaries, audio mail and audio diaries, audio note taking, security cameras, augmented reality, and body gestures.
The main classes used for capturing audio and video from media devices are:
Represents a helper class for obtaining information about available capture devices (audio or video) and requesting client user permission to access the captures from available devices.
Provides methods that work with specific audio or video captures from the associated capture device.
Describes desired and supported video format information for a video capture device, such as a webcam.
Described desired and supported formats for an audio capture device, such as a microphone.
Exposes the capture graph for video devices. Derive from this class to receive video information and to obtain the capture graph through VideoSink.CaptureSource.
Exposes the capture graph for audio devices. Derive from this class to receive audio information and to obtain the capture graph through AudioSink.CaptureSource.
When working with media devices in your Silverlight application, the first thing you need to do are get the available devices on the system. CaptureDeviceConfiguration exposes a number of static members for accomplishing this task. Once you have the devices, you can associate them with a CaptureSource.
CaptureSource is the main class used to interact with audio and video devices. Through the CaptureSource you can start and stop the audio and video devices, check the state of the devices, and capture a single video frame. A CaptureSource is associated with one AudioCaptureDevice and one VideoCaptureDevice, though it need not be associated with both. For instance, it is valid to have a CaptureSource that is only hooked into a VideoCaptureDevice.
AudioCaptureDevice and VideoCaptureDevice are the interfaces to the physical devices. Both of these classes implement CaptureDevice. CaptureDeviceConfiguration has a number of static methods for getting the audio and video devices on the system. GetAvailableAudioCaptureDevices returns a collection of all the audio devices on the systems and GetAvailableVideoCaptureDevices returns a collection of all the video devices on the system. GetDefaultAudioCaptureDevice will return the default audio device and GetDefaultVideoCaptureDevice will return the default video device. The default devices can be changed by the user in the Silverlight Configuration Settings dialog. See the section in this article on the Silverlight Configuration settings for more details.
On Mac OSX, if an application calls GetDefaultAudioCaptureDevice or GetDefaultVideoCaptureDevice and the default audio and video devices have not previously been set by the user, Silverlight will return the first device found. If a physical device is not connected to this CaptureDevice, an InvalidOperationException will be thrown when Start is called. This expectation should be caught by the application and the user can then be notified to manually set their default devices in the "Webcam/Mic" section of the Configuration Settings dialog.
The following example shows how to create a CaptureSource and associate it with the default audio and video devices on the system.
The following example show how a set the source of a ComboBox to the list of available video devices on the system. The FriendlyName of the device is used to bind to the control. An approach like this can be useful if you would like to give the user an easy way to choose the media device they would like to use. You can then listen to the SelectionChanged event for the control and set the CaptureSource to the appropriate device.
You can get the supported video and audio formats of a particular device by checking the SupportedFormats property. This method returns a collection of either VideoFormat or AudioFormat objects. The VideoFormat object specifies the FramesPerSecond, PixelFormat, PixelHeight, PixelWidth, and Stride. The AudioFormat object specifies the BitsPerSample, Channels, SamplesPerSecond, and the WaveFormat.
To display video from a video device, such as a webcam, you must have an object to render the video onto and you must be able to start and stop the video stream.
The easiest way to solve the first problem of displaying the video feed is to use a VideoBrush. Set the source of the VideoBrush to the CaptureSource. The brush can then be used to paint a rendering object, such as a Shape like a Rectangle or even a Control like a Button.
The CaptureSource object solves the second problem of interacting with the device. To start the audio and video devices associated with a CaptureSource, call the Start method. To stop the audio and video devices associated with a CaptureSource, call the Stop method.
Before a Silverlight application can play audio or video form a device, it must first be granted access by the user. To request access in your Silverlight application, call the static method RequestDeviceAccess on the CaptureDeviceConfiguration class. This method must be called in response to a user initiated event, such as a Button Click event. If RequestDeviceAccess is not called from a user initiated event, the method will always return false. If the Start method is called and device access has not been granted, an InvalidOperationException will be thrown. The AllowedDeviceAccess property on CaptureDeviceConfiguration specifies whether or not device access has been granted. For more information on media devices and privacy, see the sections in this document on Privacy and Configuration Settings.
The following example shows how to start and stop a video feed form a VideoCaptureDevice. The Start and Stop methods are called from within Button Click event handlers, which are user initiated events. Before the device is started, we request access to the device and we check that the device is not null. Before the device is stopped, we check that the capture state of the device is Started and we check that the device is not null.
Private Sub StartButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs) ' Request access to the device and verify the VideoCaptureDevice is not null. If CaptureDeviceConfiguration.RequestDeviceAccess _ AndAlso (captureSource.VideoCaptureDevice IsNot Nothing) Then Try captureSource.Start() Catch ex As InvalidOperationException ' Notify user that the webcam could not be started. MessageBox.Show("There was a problem starting the webcam " + "If using a Mac, verify default devices are set correctly. " + "Right click on a Silverlight app to access the Configuration setings.") End Try End If End Sub Private Sub StopButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs) ' Verify the VideoCaptureDevice is not null. If (captureSource.VideoCaptureDevice IsNot Nothing) Then captureSource.Stop() End If End Sub
You can capture individual frames from a video device by using the CaptureImageAsync method on the CaptureSource. This method initiates an asynchronous image capture. To be notified when then image capture is complete and to get a handle to the captured image, listen to the CaptureImageCompleted event. To be notified if an image capture failed, listen to the CaptureFailed event.
CaptureImageCompletedEventArgs is the event argument class for CaptureImageCompleted. The Result property on the argument object is the captured image. The image is returned as a WriteableBitmap. This WriteableBitmap can then be displayed, saved, or modified. Two other properties on the event arguments that may be of interest are Cancelled and Error. Cancelled is a boolean value which specifies whether the asynchronous operation was canceled. The Error property is an exception that indicates the error which occurred during the asynchronous operation. If no error occurred, Error is null.
The following example shows how to initiate an asynchronous image capture. Before the image capture is initiated, we first check that the device is not null and that the state of the device is started.
Private Sub CaptureButton_Click(ByVal sender As System.Object, ByVal e As System.Windows.RoutedEventArgs) 'Verify the VideoCaptureDevice is not null and the device is started. If (captureSource.VideoCaptureDevice IsNot Nothing) _ AndAlso (captureSource.State = CaptureState.Started) Then captureSource.CaptureImageAsync() End If End Sub
The following example creates event handlers for the CaptureImageCompleted event and the CaptureFailed event. In the CaptureImageCompleted event handler, the ImageSource property on the ImageBrush is set to the Result property.
Private Sub CaptureSource_CaptureImageCompleted(ByVal sender As Object, ByVal e As CaptureImageCompletedEventArgs) ' Set the ImageBrush source (defined in XAML) to the Result property capturedImage.ImageSource = e.Result End Sub Private Sub CaptureSource_CaptureFailed(ByVal sender As Object, ByVal e As ExceptionRoutedEventArgs) ' Handle the failure. End Sub
To capture and play back audio, you will need to use an AudioSink object. This object gives you access to the raw audio stream coming from the audio device. This stream can then be encoded into a common format. AudioSink is an abstract class so you will need to create a class that implements AudioSink. There are four virtual callbacks you need to override: OnCaptureStarted, OnCaptureStopped, OnFormatChange, and OnSamples.
OnCaptureStarted and OnCaptureStopped occur when the audio device is started and stopped. These notifications can be useful for creating sentinels in your application to block certain actions when capture is occurring.
OnFormatChange occurs when the devices detects the audio format has changed. OnFormatChange is invoked at least one time for the first capture by the sink. Therefore, you can use OnFormatChange as the definitive determination of the audio format instead of using DesiredFormat or SupportedFormats. It is possible for the AudioFormat associated with the AudioSink to change between the time that the device was first connected and the time of the capture. Listening to OnFormatChange is the only way to be certain of the true AudioFormat.
OnSamples is invoked when the audio device captures a complete audio sample. The arguments passed into this method are the sample time (in hundreds of nanoseconds), the sample duration (in hundred of nanoseconds), and the sample data as a byte array. The interval at which OnSamples is called depends on the AudioFrameSize value for the AudioCaptureDevice. The AudioFormat specifies the bits per second, the number of channels, and the wave format. This information can then be use to convert the sample data into the desired format.
Just as you can get access to the raw audio from the audio device, you can also get the raw video form the video device through a VideoSink. VideoSink is an abstract class so you will need to create a class that implements VideoSink. The structure of the VideoSink is similar to the AudioSink. You will need to override the same four virtual callbacks: OnCaptureStarted, OnCaptureStopped, OnFormatChange, and OnSample.
These methods behave in a similar manner as their AudioSink counter parts.
In OnSample you can use the information from the VideoFormat, which can be obtained by listening to the OnFormatChange method, to convert the bit stream into the desired video format. The VideoFormat specifies the pixel format, the height of the frame in pixels, the width of the frame in pixels, and the stride.
In order for Silverlight to gain access to audio and video devices, the user of the application must first grant access to the devices. The user is prompted with a privacy dialog asking if they want to allow camera and microphone access. If the user agrees to this dialog, then the Silverlight application is granted access to the devices; otherwise, the application cannot access the devices. On the privacy dialog there is a check box for remembering the choice the user made. Access rights can be changed, removed, or added for sites in the Silverlight Configuration dialog (right click on a running Silverlight application).
The Silverlight Configuration Settings dialog has a few options that pertain to audio and video devices. To open the configuration settings, right click on any running Silverlight application and choose the Silverlight node. For more information on the Silverlight Configuration Settings, see the Microsoft Silverlight Configuration Dialog Box topic.
The Configuration Settings has a node for webcams and microphones. From this dialog you can change the default video device and the default audio device. These settings are independent of each other, so you could choose video from one device and audio from another device.
From the Permission tab on the Configuration Settings dialog you can allow, deny, or remove access to a particular site. If for example, a user granted webcam access to a particular site they could remove that access from this dialog.