Multimedia

Develop an app that’s ready to help users both create and play back exciting media experiences. This is now easier to do in Windows 8.1, with new properties for capturing media in different environments, a simple way to add playback controls to apps that use XAMLExtensible Application Markup Language (XAML), and native support for Media Source Extensions (MSE) for HTML5 playback.

Updated in Windows 8.1

Video processing feature updates

[Get the Media Hub and MediaStreamSource Sample now.]

Windows 8.1 introduces some new updates and support related to video formats, codecs, and processing.

Support for CFF

Microsoft Media Foundation added support for Common File Format (CFF) to its MP4/fMP4 implementations. CFF is an audio/video container format based on the ISO Base Media File Format.

New XVP mode

The Media Foundation Transcode Video Processor (XVP) added the MrfCrf444 mode, which always runs in software and does not use DirectX Video Acceleration (DXVA) hardware acceleration. This ensures consistent results across platforms, independent of underlying hardware. Because the MrfCrf444 mode executes entirely in software, the transcode operation may take longer and use more power. This new mode forces the use of larger filters, uses full precision for color conversions, prefers higher-quality color formats, and enables a high quality deinterlacer.

You can use this in your app by setting the MediaTrancoder.VideoProcessingAlgorithm to MediaVideoProcessingAlgorithm.mrfCrf444.

No code example is currently available or this language may not be supported.

Support for extended range YUV in Media Foundation

In Windows 8, the Media Foundation pipeline defaulted to studio swing YUV range [Y: 16-235] and did not provide the ability to signal if the input was in extended range YUV [Y: 0-255]. The Media Foundation pipeline now has the ability to signal extended range to the graphics processing unit (GPU) hardware. This makes the entire Media Foundation pipeline, including the MJPEG decoder, extended-range aware.

The MF_MT_VIDEO_NOMINAL_RANGE attribute is used to specify studio or extended YUV range for the content.

Media Stream Source

Windows 8.1 introduces a new generic media source for Windows Store apps called Media Stream Source (MSS). The APIs for this technology are in the Windows.Media.Core namespace, and the main media-source class is MediaStreamSource. MSS enables apps to send compressed and uncompressed audio and video samples to the media pipeline for playback, transcoding, and streaming. Media samples can be dynamically generated by the app, or de-multiplexed from a stream or files. This flexibility enables apps to more easily extend platform support for new media formats or solve complex problems, such as adaptive streaming or implementing a custom media source in managed code.

The MediaStreamSource API are very similar to the Microsoft Silverlight APIs of the same name. So developers who are familiar with the Silverlight versions of these APIs can apply much of their knowledge to build Windows Store apps using MSS.

MSS can be used with audio and video objects in Windows Store apps using JavaScript, MediaElement objects in Windows Store apps using C++, C#, or Visual Basic, and the MediaTranscoder.

Here are some of the main MSS APIs. The order reflects the basic flow of how MSS functions. You'll notice that the MediaStreamSource sends request objects to the app through event arguments. These request objects enable the app to interact with the MediaStreamSource and pass data back to it.

API Description
MediaStreamSource Represents a media source that delivers media samples directly to the media pipeline. MediaStreamSource consumes MediaStreamSample objects that are provided by the app.
MediaStreamSample Represents a media sample used by the MediaStreamSource.
MediaStreamSource.Starting (event)

The MediaStreamSource uses this event to notify the app that it is ready to start processing media data.

MediaStreamSourceStartingRequest

Represents a request from the MediaStreamSource indicating that it is ready to start processing media data.

Apps should reply as soon as possible to this request by calling SetActualStartPosition. If an app needs to delay the MediaStreamSource from processing data, it can get an asynchronous deferral by calling MediaStreamSourceStartingRequest.GetDeferral. When the app is ready for the MediaStreamSource to start, it calls Complete on the deferral object.

The starting request is accessed through the MediaStreamSourceStartingEventArgs that are passed to the MediaStreamSource.Starting event handler.

MediaStreamSource.SampleRequested (event)

The MediaStreamSource uses this event to notify the app that it is ready for a MediaStreamSample.

Apps are required to register a handler for this event.

MediaStreamSourceSampleRequest

Represents a request from the MediaStreamSource for a new media sample. Setting the Sample property to the new MediaStreamSample triggers the MediaStreamSource to retrieve the media sample and continue processing the media data.

Apps should reply as soon as possible to this request. If an app needs time before sending the MediaStreamSample, it can get an asynchronous deferral by calling MediaStreamSourceSampleRequest.GetDeferral. When the app is finished with the deferral, it calls Complete on the deferral object.

The sample request is accessed through the MediaStreamSourceSampleRequestedEventArgs that are passed to the MediaStreamSource.SampleRequest event handler.

The app indicates it has reached the end of the stream by responding to a MediaStreamSourceSampleRequest without providing a MediaStreamSample, or by assigning the MediaStreamSourceSampleRequest.Sample property to null.

MediaStreamSource.Closed (event)

The MediaStreamSource uses this event to notify the app that it has shut down.

MediaStreamSourceClosedRequest

Represents a request from the MediaStreamSource indicating that it has closed.

The close request is accessed through the MediaStreamSourceClosedEventArgs that are passed to the MediaStreamSource.Closed event handler.

MediaElement.SetMediaStreamSource Sets the source of the MediaElement to a MediaStreamSource.

 

Get the MediaStreamSource Sample for a detailed example of using MSS.

Audio and video capture updates

[Get the Media Capture and Audio effects discovery samples now.]

Windows 8.1 introduces a host of feature updates for audio and video capture.

Photo sequence mode

Photo sequence mode takes a continuous sequence of photos. You configure this mode by using the LowLagPhotoSequenceControl class, which you access through the MediaCapture.VideoDeviceController property.

Use the LowLagPhotoSequenceControl.Supported property to find out if a device supports photo sequence mode.

A key scenario for photo sequence mode is getting photos in the past—that is, before the user takes a photo. You can use PhotoCapturedEventArgs.CaptureTimeOffset to tell whether a frame was in the future or past. A value greater than 0 indicates the future and less than 0 indicates the past.

To initiate this mode, call MediaCapture.PrepareLowLagPhotoSequenceCaptureAsync. To start capturing photos, call StartAsync. Your app will continue to receive photos from the device until the operation is stopped. To stop capturing photos, call StopAsync.

If your app wants to set how many frames it gets per second, it can use LowLagPhotoSequence.PhotosPerSecondLimit. This can be useful in situations where the sensor on the device can handle 30 frames per second (fps), but the app needs only 4 fps.

Thumbnail support

Thumbnails are supported for single photos and for photo-sequence operations through the LowLagPhotoControl and LowLagPhotoSequenceControl classes, which you access through the MediaCapture.VideoDeviceController property.

To enable thumbnails, set the ThumbnailEnabled property to true. You can set the thumbnail size you want by using DesiredThumbnailSize and set the format with ThumbnailFormat.

Scene Mode

Scene Mode tunes the post-processing of captured frames to better match the scene currently being recorded. You configure this mode by using SceneModeControl, which you access through the MediaCapture.VideoDeviceController property.

The SupportedModes property indicates which Scene Modes (like Snow, Night, and Sport) are supported by a device. To select a mode, call SceneModeControl.SetValueAsync.

Torch Mode

Torch Mode uses the torch LED on a device. You configure this mode by using TorchControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the TorchControl.Supported property. If it's supported, you can use Torch Mode in capture apps, or in non-capture apps to do things like brighten a room.

Flash Mode

Flash Mode uses the flash on a device. You configure this mode by using FlashControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the FlashControl.Supported property. If it's supported, you can use Flash Mode in a photo app to handle low-light conditions or for RedEyeReduction.

White Balance

White Balance gives your app additional control over the white-balance settings on a device. You can configure these levels by using WhiteBalanceControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the WhiteBalanceControl.Supported property. If it's supported, you can use one of the ColorTemperaturePreset values by calling SetPresetAsync. Or call SetValueAsync to set a specific Value between the Min and Max.

Exposure Mode

Exposure Mode gives your app additional control over the exposure settings on a device. You configure this mode by using ExposureControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the ExposureControl.Supported property. If it's supported, your app can set exposure values by calling SetValueAsync. To turn auto exposure on, call SetAutoAsync.

Exposure Compensation Mode

Exposure Compensation Mode gives your app control over the exposure levels of captured photos. You configure this mode by using ExposureCompensationControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the ExposureCompensationControl.Supported property. If it does, your app can set exposure values by calling SetValueAsync.

Focus Mode

Focus Mode gives your app additional control over the focus settings on a device. You configure this mode by using FocusControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the FocusControl.Supported property. If it does, you can use one of the FocusPreset values by calling SetPresetAsync. Or call SetValueAsync to set a specific Value between the Min and Max.

ISO Mode

ISO Mode enables your app to manage the ISO speed settings on a device. In low-light conditions, you may prefer a higher noise level in return for higher overall brightness. The ISO speed control lets your app adjust this tradeoff. You configure this mode by using IsoSpeedControl, which you access through the MediaCapture.VideoDeviceController property.

You can find out whether a device supports this control by checking the IsoSpeedControl.Supported property. If it does, use SetPresetAsync to set the ISO level to one of the IsoSpeedPreset values. SupportedPresets lists the preset values that the device supports.

Region of Interest

Region of Interest lets your app specify the RegionOfInterest in a preview. You configure this setting by using RegionsOfInterestControl, which you access through MediaCapture.VideoDeviceController.

Use this setting to specify the area of preview for which to compute things like focus and exposure levels. Region of Interest enables scenarios like "tap to focus."

Field of View

Field of View lets your app compute the field of view and camera angle with the MediaCaptureSettings.PitchOffsetDegrees, MediaCaptureSettings.Horizontal35mmEquivalentFocalLength, and MediaCaptureSettings.Vertical35mmEquivalentFocalLength settings.

Take photo while recording video

Your app can check the MediaCaptureSettings.ConcurrentRecordAndPhotoSupported settings to find out whether the device allows taking a photo while recording video.

Determine whether camera is required to play sound

The MediaCaptureSettings.CameraSoundRequiredForRegion setting indicates whether the operating system is required to play a sound when the camera takes a picture. Your app can use this info to decide whether a sound must be played.

Capture device optimization

The VideoDeviceController.PrimaryUse property indicates whether the capture device is being used primarily for photos or videos. This enables the system to optimize for performance and quality.

Record uncompressed AVI and WAV

Your app can record the uncompressed audio stream from the microphone by setting MediaCaptureInitializationSettings.AudioProcessing to Raw.

Audio effects discovery

Your app can discover the audio processing chain on an endpoint. You can use this info to configure the audio processing that is done in your app. The Windows.Media.Effects namespace contains the AudioEffect, AudioCaptureEffectsManager, AudioEffectsManager, and AudioRenderEffectsManager classes.

To create an AudioCaptureEffectsManager object, call AudioEffectsManager.CreateAudioCaptureEffectsManager. To create an AudioRenderEffectsManager object, call AudioEffectsManager.CreateAudioRenderEffectsManager.

XAML MediaElement updates

[Get the XAML media playback sample now.]

The MediaElement control has been extended to support built-in transport controls and full-window rendering.

The MediaElement.AreTransportControlsEnabled property enables or disables built-in transport controls, so you no longer need to wire up your own buttons to do things like play, stop, pause, or scrubbing. This API works like the controls property on the HTML5 video and audio elements.

The MediaElement.IsFullWindow property enables or disables full-window rendering.

Here's some code that shows how to enable the built-in transport controls.



<MediaElement Name="media" 
              Source="testVideo.mp4" 
              AreTransportControlsEnabled="True" />


MediaElement media = new MediaElement();
media.AreTransportControlsEnabled = true;

And here's some code that shows how to enable full-window video rendering.


<MediaElement Name="media"
              Source="testVideo.mp4" 
              IsFullWindow="True" />


MediaElement media = new MediaElement();
media.IsFullWindow = true;

HTML5 video and audio playback

[Get the HTML media playback sample now.]

Windows 8.1 introduces a number of updates to audio and video media playback, including support for the World Wide Web Consortium (W3C) HTML5 Media Source Extensions (MSE), support for the W3C HTML5 Encrypted Media Extensions (EME), and updates to the XAML MediaElement control.

Media Source Extensions (MSE) for HTML5 video and audio.

HTML5 Media Source Extensions (MSE) is a W3C standard that extends the video and audio elements to enable dynamically changing the media stream without the use of plug-ins. This supports scenarios like adaptive media streaming, live streaming, splicing videos, and video editing.

Custom adaptive streaming solutions can be built on MSE. Adaptive streaming solutions dynamically change the bit rate of the media stream in order to provide the highest quality streaming experience. They attempt to determine the optimal bit rate based on constraints such as network bandwidth and CPU use. For example, if network bandwidth drops, you can use MSE to switch the video stream to a lower bit rate. This would reduce the quality of the image, but would enable the video to continue playing without interruption.

One use for dynamically splicing videos together is to insert ads during media playback. This means you do not have to put ad clips in the media files beforehand, but can instead insert a clip in the media stream during playback.

Here are some of the important elements of the MSE API.

API Description
MediaSource The media object that is attached to the HTML media element. MediaSource contains the media data that is specified by one or more SourceBuffer objects.
MediaSource.readyState Indicates the current state of the MediaSource and whether it is attached to a media element.
SourceBuffer The buffer used to hold the media data.
SourceBufferList A collection of SourceBuffer objects.
MediaSource.endOfStream Indicates the cause of an end of the stream.
MediaSource.sourceBuffers The collection of SourceBuffer objects that contain the media data for the MediaSource.
createObjectURL Creates a URL to the MediaSource object. The MediaSource can be attached to the video or audio by setting the src property to this URL.

 

Encrypted Media Extensions (EME) for HTML5 video and audio elements

HTML5 Encrypted Media Extensions (EME) is a W3C standard that extends the video and audio elements to enable Digital Rights Management (DRM) protected content without using plug-ins. EME is supported in Internet Explorer 11 and in Windows Store apps using JavaScript running on Windows 8.1.

Here are some of the main APIs used with EME.

API Description
MediaKeys The keys used to decrypt and encrypt the media data.
MediaKeySession The playback session that the keys are associated with.
MediaKeys.createSession Creates the MediaKeySession associated with the MediaKeys.
keySystem Attribute on the media element that specifies the Key System. The Key System identifies either the content-protection provider or the decryption mechanism.
MediaKeyError Specifies the types of errors associated with MediaKeys.
MediaKeyMessageEvent Specifies the types of MediaKeys-related events.

 

Play To feature updates

[Get the Media Play To and SystemMediaTransportControls samples now.]

Windows 8.1 introduces a number of updates to improve the Play To experience.

Programmatically launch the Play To UI

New UI guidance for Windows 8.1 encourages app developers to provide an in-app button, using a standard Play To glyph, to programmatically display the Play To flyout, and not to require the user to bring up the UI by swiping the Devices charm. Use ShowPlayToUI to display the flyout.

Here's some code that shows how to display the Play To UI.

No code example is currently available or this language may not be supported.

Play To by reference

Play To by reference enables webpages and Windows Store apps to play DRM-protected content. The app specifies a preferred Uniform Resource Identifier (URI) to be sent to the Play To target device. This URI can point to a cloud-based media server. If the preferredSourceUri is present, the Play To target device can stream content directly from the cloud, and that content can be DRM protected. (Contrast the alternative—streaming content from the Windows 8.1 machine—where the content must be unprotected.)

Here's some code that shows how to set up Play To by reference.



// video is a MediaElement defined in XAML.
using Windows.Foundation;

video.Source = new Uri("http://www.contoso.com/videos/video.mp4"); 
video.PlayToSource.PreferredSourceUri = 
   new Uri("http://www.contoso.com/catalogid=1234"); 




<video src="http://www.contoso.com/videos/video.mp4" 
   x-ms-playToPreferredSourceUri="http://www.contoso.com/catalogid=1234" /> 


auto video = ref new Windows::UI::Xaml::Controls::MediaElement(); 

video->Source = 
   ref new Windows::Foundation::Uri("http://www.contoso.com/videos/video.mp4"); 

video->PlaytoSource->PreferredSourceUri = 
   ref new Windows::Foundation::Uri("http://www.contoso.com/catalogid=1234"); 


Adobe Flash audio and video support

Play To now supports playing unprotected Adobe Flash audio and video from Internet Explorer in the new Windows UI on a Digital Media Receiver (DMR).

Support for Media Source Extensions

Media Source Extensions (MSE) is a new W3C standard that will become the backbone of high-quality, premium, power-efficient, feature-length video delivery using only a web browser. MSE does not require a plug-in. Play To now supports playing unprotected MSE content from Windows Store app using JavaScript and websites in Internet Explorer in the Windows UI.

Video to audio-only devices

The Play To UI now supports selecting an audio-only receiver as a remote endpoint for video content. The video content is demixed and just the audio component is sent to the device.

Media transport controls UI

In Windows 8.1, you can use the new SystemMediaTransportControls instead of the MediaControl class.

The media transport controls UI, the UI you see when you press volume up or down, can now be used as a remote control for playback on a Play To device.

The transport controls can be displayed and used to control local or remote playback while the source app is in the background or while the device screen is locked. All Windows Store apps get a default implementation that includes:

  • Hooking up basic transport controls, such as play, pause, and seek

  • The app name in place of song title

  • The app icon in place of album art

Your app can improve on the default implementation by:

  • Hooking up advanced transport controls, such as next-track and previous-track functionality

  • Providing the song title

  • Providing the album art

  • Providing the album title

Devices charm UI updates

The Devices charm is now organized hierarchically and puts devices in these categories:

  • Play

  • Print

  • Project

  • Tap & Send (if the device hardware supports it)

This helps reduce confusion for users about what they can expect to find in the Devices charm. Even when there are no devices currently available, these sections are visible.

Digital Media Receiver persistence

The most recently selected Digital Media Receiver, along with its connection status, is now highlighted on the Device charm.

Improved connection/disconnection flow, and better error reporting

The Play To UI now supports an improved connection and disconnection flow.

While a Play To connection is being established with a device, a new UI shows the message "Connecting" and the Play To flyout remains visible until the connection has been made. If the connection fails, a connection error is reported to the source app.

The Disconnect button has been removed. Instead, to end remote playback and resume playback locally, users now select This PC, which is listed alongside other devices in the Play To UI.

 

 

Show:
© 2014 Microsoft. All rights reserved.