Develop an app that’s ready to help users both create and play back exciting media experiences. This is now easier to do in Windows 8.1, with new properties for capturing media in different environments, a simple way to add playback controls to apps that use XAMLExtensible Application Markup Language (XAML), and native support for Media Source Extensions (MSE) for HTML5 playback.
- Video processing feature
- Audio and video capture feature
- XAML MediaElement
- HTML5 video and audio playback
- Play To feature
Windows 8.1 introduces some new updates and support related to video formats, codecs, and processing.
Microsoft Media Foundation added support for Common File Format (CFF) to its MP4/fMP4 implementations. CFF is an audio/video container format based on the ISO Base Media File Format.
The Media Foundation Transcode Video Processor (XVP) added the MrfCrf444 mode, which always runs in software and does not use DirectX Video Acceleration (DXVA) hardware acceleration. This ensures consistent results across platforms, independent of underlying hardware. Because the MrfCrf444 mode executes entirely in software, the transcode operation may take longer and use more power. This new mode forces the use of larger filters, uses full precision for color conversions, prefers higher-quality color formats, and enables a high quality deinterlacer.
// sourceFile and destFile are IStorageFile objects defined elsewhere. var profile = Windows.Media.MediaProperties.MediaEncodingProfile.createMp4 (Windows.Media.MediaProperties.VideoEncodingQuality.hd720p); var transcoder = new Windows.Media.Transcoding.MediaTranscoder(); transcoder.videoProcessingAlgorithm = Windows.Media.Transcoding.MediaVideoProcessingAlgorithm.mrfCrf444; transcoder.prepareFileTranscodeAsync(sourceFile, destFile, profile);
In Windows 8, the Media Foundation pipeline defaulted to studio swing YUV range [Y: 16-235] and did not provide the ability to signal if the input was in extended range YUV [Y: 0-255]. The Media Foundation pipeline now has the ability to signal extended range to the graphics processing unit (GPU) hardware. This makes the entire Media Foundation pipeline, including the MJPEG decoder, extended-range aware.
The MF_MT_VIDEO_NOMINAL_RANGE attribute is used to specify studio or extended YUV range for the content.
Windows 8.1 introduces a new generic media source for Windows Store apps called Media Stream Source (MSS). The APIs for this technology are in the Windows.Media.Core namespace, and the main media-source class is MediaStreamSource. MSS enables apps to send compressed and uncompressed audio and video samples to the media pipeline for playback, transcoding, and streaming. Media samples can be dynamically generated by the app, or de-multiplexed from a stream or files. This flexibility enables apps to more easily extend platform support for new media formats or solve complex problems, such as adaptive streaming or implementing a custom media source in managed code.
The MediaStreamSource API are very similar to the Microsoft Silverlight APIs of the same name. So developers who are familiar with the Silverlight versions of these APIs can apply much of their knowledge to build Windows Store apps using MSS.
Here are some of the main MSS APIs. The order reflects the basic flow of how MSS functions. You'll notice that the MediaStreamSource sends request objects to the app through event arguments. These request objects enable the app to interact with the MediaStreamSource and pass data back to it.
|MediaStreamSource||Represents a media source that delivers media samples directly to the media pipeline. MediaStreamSource consumes MediaStreamSample objects that are provided by the app.|
|MediaStreamSample||Represents a media sample used by the MediaStreamSource.|
The MediaStreamSource uses this event to notify the app that it is ready to start processing media data.
Represents a request from the MediaStreamSource indicating that it is ready to start processing media data.
Apps should reply as soon as possible to this request by calling SetActualStartPosition. If an app needs to delay the MediaStreamSource from processing data, it can get an asynchronous deferral by calling MediaStreamSourceStartingRequest.GetDeferral. When the app is ready for the MediaStreamSource to start, it calls Complete on the deferral object.
Apps are required to register a handler for this event.
Represents a request from the MediaStreamSource for a new media sample. Setting the Sample property to the new MediaStreamSample triggers the MediaStreamSource to retrieve the media sample and continue processing the media data.
Apps should reply as soon as possible to this request. If an app needs time before sending the MediaStreamSample, it can get an asynchronous deferral by calling MediaStreamSourceSampleRequest.GetDeferral. When the app is finished with the deferral, it calls Complete on the deferral object.
The app indicates it has reached the end of the stream by responding to a MediaStreamSourceSampleRequest without providing a MediaStreamSample, or by assigning the MediaStreamSourceSampleRequest.Sample property to null.
The MediaStreamSource uses this event to notify the app that it has shut down.
Represents a request from the MediaStreamSource indicating that it has closed.
|MediaElement.SetMediaStreamSource||Sets the source of the MediaElement to a MediaStreamSource.|
Get the MediaStreamSource Sample for a detailed example of using MSS.
Windows 8.1 introduces a host of feature updates for audio and video capture.
Photo sequence mode takes a continuous sequence of photos. You configure this mode by using the LowLagPhotoSequenceControl class, which you access through the MediaCapture.VideoDeviceController property.
Use the LowLagPhotoSequenceControl.Supported property to find out if a device supports photo sequence mode.
A key scenario for photo sequence mode is getting photos in the past—that is, before the user takes a photo. You can use PhotoCapturedEventArgs.CaptureTimeOffset to tell whether a frame was in the future or past. A value greater than 0 indicates the future and less than 0 indicates the past.
To initiate this mode, call MediaCapture.PrepareLowLagPhotoSequenceCaptureAsync. To start capturing photos, call StartAsync. Your app will continue to receive photos from the device until the operation is stopped. To stop capturing photos, call StopAsync.
If your app wants to set how many frames it gets per second, it can use LowLagPhotoSequence.PhotosPerSecondLimit. This can be useful in situations where the sensor on the device can handle 30 frames per second (fps), but the app needs only 4 fps.
Thumbnails are supported for single photos and for photo-sequence operations through the LowLagPhotoControl and LowLagPhotoSequenceControl classes, which you access through the MediaCapture.VideoDeviceController property.
Scene Mode tunes the post-processing of captured frames to better match the scene currently being recorded. You configure this mode by using SceneModeControl, which you access through the MediaCapture.VideoDeviceController property.
You can find out whether a device supports this control by checking the TorchControl.Supported property. If it's supported, you can use Torch Mode in capture apps, or in non-capture apps to do things like brighten a room.
You can find out whether a device supports this control by checking the FlashControl.Supported property. If it's supported, you can use Flash Mode in a photo app to handle low-light conditions or for RedEyeReduction.
White Balance gives your app additional control over the white-balance settings on a device. You can configure these levels by using WhiteBalanceControl, which you access through the MediaCapture.VideoDeviceController property.
You can find out whether a device supports this control by checking the WhiteBalanceControl.Supported property. If it's supported, you can use one of the ColorTemperaturePreset values by calling SetPresetAsync. Or call SetValueAsync to set a specific Value between the Min and Max.
Exposure Mode gives your app additional control over the exposure settings on a device. You configure this mode by using ExposureControl, which you access through the MediaCapture.VideoDeviceController property.
You can find out whether a device supports this control by checking the ExposureControl.Supported property. If it's supported, your app can set exposure values by calling SetValueAsync. To turn auto exposure on, call SetAutoAsync.
Exposure Compensation Mode gives your app control over the exposure levels of captured photos. You configure this mode by using ExposureCompensationControl, which you access through the MediaCapture.VideoDeviceController property.
Focus Mode gives your app additional control over the focus settings on a device. You configure this mode by using FocusControl, which you access through the MediaCapture.VideoDeviceController property.
You can find out whether a device supports this control by checking the FocusControl.Supported property. If it does, you can use one of the FocusPreset values by calling SetPresetAsync. Or call SetValueAsync to set a specific Value between the Min and Max.
ISO Mode enables your app to manage the ISO speed settings on a device. In low-light conditions, you may prefer a higher noise level in return for higher overall brightness. The ISO speed control lets your app adjust this tradeoff. You configure this mode by using IsoSpeedControl, which you access through the MediaCapture.VideoDeviceController property.
You can find out whether a device supports this control by checking the IsoSpeedControl.Supported property. If it does, use SetPresetAsync to set the ISO level to one of the IsoSpeedPreset values. SupportedPresets lists the preset values that the device supports.
Use this setting to specify the area of preview for which to compute things like focus and exposure levels. Region of Interest enables scenarios like "tap to focus."
Field of View lets your app compute the field of view and camera angle with the MediaCaptureSettings.PitchOffsetDegrees, MediaCaptureSettings.Horizontal35mmEquivalentFocalLength, and MediaCaptureSettings.Vertical35mmEquivalentFocalLength settings.
Your app can check the MediaCaptureSettings.ConcurrentRecordAndPhotoSupported settings to find out whether the device allows taking a photo while recording video.
The MediaCaptureSettings.CameraSoundRequiredForRegion setting indicates whether the operating system is required to play a sound when the camera takes a picture. Your app can use this info to decide whether a sound must be played.
The VideoDeviceController.PrimaryUse property indicates whether the capture device is being used primarily for photos or videos. This enables the system to optimize for performance and quality.
Your app can record the uncompressed audio stream from the microphone by setting MediaCaptureInitializationSettings.AudioProcessing to Raw.
Your app can discover the audio processing chain on an endpoint. You can use this info to configure the audio processing that is done in your app. The Windows.Media.Effects namespace contains the AudioEffect, AudioCaptureEffectsManager, AudioEffectsManager, and AudioRenderEffectsManager classes.
To create an AudioCaptureEffectsManager object, call AudioEffectsManager.CreateAudioCaptureEffectsManager. To create an AudioRenderEffectsManager object, call AudioEffectsManager.CreateAudioRenderEffectsManager.
[Get the XAML media playback sample now.]
The MediaElement control has been extended to support built-in transport controls and full-window rendering.
The MediaElement.AreTransportControlsEnabled property enables or disables built-in transport controls, so you no longer need to wire up your own buttons to do things like play, stop, pause, or scrubbing. This API works like the controls property on the HTML5 video and audio elements.
The MediaElement.IsFullWindow property enables or disables full-window rendering.
Here's some code that shows how to enable the built-in transport controls.
And here's some code that shows how to enable full-window video rendering.
[Get the HTML media playback sample now.]
Windows 8.1 introduces a number of updates to audio and video media playback, including support for the World Wide Web Consortium (W3C) HTML5 Media Source Extensions (MSE), support for the W3C HTML5 Encrypted Media Extensions (EME), and updates to the XAML MediaElement control.
HTML5 Media Source Extensions (MSE) is a W3C standard that extends the video and audio elements to enable dynamically changing the media stream without the use of plug-ins. This supports scenarios like adaptive media streaming, live streaming, splicing videos, and video editing.
Custom adaptive streaming solutions can be built on MSE. Adaptive streaming solutions dynamically change the bit rate of the media stream in order to provide the highest quality streaming experience. They attempt to determine the optimal bit rate based on constraints such as network bandwidth and CPU use. For example, if network bandwidth drops, you can use MSE to switch the video stream to a lower bit rate. This would reduce the quality of the image, but would enable the video to continue playing without interruption.
One use for dynamically splicing videos together is to insert ads during media playback. This means you do not have to put ad clips in the media files beforehand, but can instead insert a clip in the media stream during playback.
Here are some of the important elements of the MSE API.
|MediaSource||The media object that is attached to the HTML media element. MediaSource contains the media data that is specified by one or more SourceBuffer objects.|
|MediaSource.readyState||Indicates the current state of the MediaSource and whether it is attached to a media element.|
|SourceBuffer||The buffer used to hold the media data.|
|SourceBufferList||A collection of SourceBuffer objects.|
|MediaSource.endOfStream||Indicates the cause of an end of the stream.|
|MediaSource.sourceBuffers||The collection of SourceBuffer objects that contain the media data for the MediaSource.|
|createObjectURL||Creates a URL to the MediaSource object. The MediaSource can be attached to the video or audio by setting the src property to this URL.|
Here are some of the main APIs used with EME.
|MediaKeys||The keys used to decrypt and encrypt the media data.|
|MediaKeySession||The playback session that the keys are associated with.|
|MediaKeys.createSession||Creates the MediaKeySession associated with the MediaKeys.|
|keySystem||Attribute on the media element that specifies the Key System. The Key System identifies either the content-protection provider or the decryption mechanism.|
|MediaKeyError||Specifies the types of errors associated with MediaKeys.|
|MediaKeyMessageEvent||Specifies the types of MediaKeys-related events.|
Windows 8.1 introduces a number of updates to improve the Play To experience.
New UI guidance for Windows 8.1 encourages app developers to provide an in-app button, using a standard Play To glyph, to programmatically display the Play To flyout, and not to require the user to bring up the UI by swiping the Devices charm. Use ShowPlayToUI to display the flyout.
Here's some code that shows how to display the Play To UI.
Play To by reference enables webpages and Windows Store apps to play DRM-protected content. The app specifies a preferred Uniform Resource Identifier (URI) to be sent to the Play To target device. This URI can point to a cloud-based media server. If the preferredSourceUri is present, the Play To target device can stream content directly from the cloud, and that content can be DRM protected. (Contrast the alternative—streaming content from the Windows 8.1 machine—where the content must be unprotected.)
Here's some code that shows how to set up Play To by reference.
// video is a MediaElement defined in XAML. using Windows.Foundation; video.Source = new Uri("http://www.contoso.com/videos/video.mp4"); video.PlayToSource.PreferredSourceUri = new Uri("http://www.contoso.com/catalogid=1234");
<video src="http://www.contoso.com/videos/video.mp4" x-ms-playToPreferredSourceUri="http://www.contoso.com/catalogid=1234" />
var video = document.createElement('video'); document.body.appendChild(video); video.src = "http://www.contoso.com/videos/video.mp4"; video.msPlayToPreferredSourceUri = "http://www.contoso.com/catalogid=1234";
Play To now supports playing unprotected Adobe Flash audio and video from Internet Explorer in the new Windows UI on a Digital Media Receiver (DMR).
The Play To UI now supports selecting an audio-only receiver as a remote endpoint for video content. The video content is demixed and just the audio component is sent to the device.
The media transport controls UI, the UI you see when you press volume up or down, can now be used as a remote control for playback on a Play To device.
The transport controls can be displayed and used to control local or remote playback while the source app is in the background or while the device screen is locked. All Windows Store apps get a default implementation that includes:
Hooking up basic transport controls, such as play, pause, and seek
The app name in place of song title
The app icon in place of album art
Your app can improve on the default implementation by:
Hooking up advanced transport controls, such as next-track and previous-track functionality
Providing the song title
Providing the album art
Providing the album title
The Devices charm is now organized hierarchically and puts devices in these categories:
Tap & Send (if the device hardware supports it)
This helps reduce confusion for users about what they can expect to find in the Devices charm. Even when there are no devices currently available, these sections are visible.
The most recently selected Digital Media Receiver, along with its connection status, is now highlighted on the Device charm.
The Play To UI now supports an improved connection and disconnection flow.
While a Play To connection is being established with a device, a new UI shows the message "Connecting" and the Play To flyout remains visible until the connection has been made. If the connection fails, a connection error is reported to the source app.
The Disconnect button has been removed. Instead, to end remote playback and resume playback locally, users now select This PC, which is listed alongside other devices in the Play To UI.