Video Effect Introduction

This article describes how to apply effects to a real-time video with the ability to record video with the effects applied.

The VideoEffect class allows you to process frames from a video stream and record the processed sample.
The Lumia Imaging SDK also contains an IImageProvider called CameraPreviewImageSource, which allows you to process frames from a video stream before capturing a final image to process. However, unless you were to provide your own video encoder, you cannot record the video being output by the CameraPreviewImageSource.

WinRT/WinPRT provides a MediaCapture class which provides video preview, image capture, and video record capabilities. The preview can be output to a XAML element CaptureElement. Using the new VideoEffect class allows developer to preview or record output from the MediaCapture class. The MediaCapture class will support Media Foundation Transforms through the AddEffectAsync method. The VideoEffect is now provided by Lumia imaging SDK.

Note: NV12 and YUY2 are the video formats supported by the VideoEffect class.

Using the VideoEffect in a C# Universal App

In order to use the VideoEffect, the VideoEffect must be registered.
This is done in the Manifest file in an XML editor. Right-click on the Package.appxmanifest file in the WP8.1 app, select "Open with...", and choose an XML editor. In between Applications and Capabilities, insert:

<Extensions>
   <Extension Category="windows.activatableClass.inProcessServer">
      <InProcessServer>
        <Path>Lumia.Imaging.dll</Path>
        <ActivatableClass ActivatableClassId="Lumia.Imaging.VideoEffect" ThreadingModel="both" />
      </InProcessServer>
    </Extension>
</Extensions>

MediaElement

To setup the page for viewing video from a file, simply place the MediaElement somewhere on the page and set its source.

   var grayscaleEffect = new GrayscaleEffect();
   var m_configurationPropertySet = new PropertySet();
   configuration = new VideoEffectConfiguration((IImageConsumer)grayscaleEffect, grayscaleEffect);           

   m_configurationPropertySet.Add(new KeyValuePair<string, object>("Configuration", configuration)); 

   m_MediaElement.Stop();
   m_MediaElement.RemoveAllEffects();
   m_MediaElement.AddVideoEffect("Lumia.Imaging.VideoEffect", false, propertySet);
   m_MediaElement.Play(); 

MediaCapture & CaptureElement

To set up the page for previewing video, simply place the CaptureElement somewhere on the page and set its source to the instance of MediaCapture generated.

Warning: Failing to shutdown the MediaCapture instance after video previewing has been started can result in the mobile device locking up!

Capture Start

Step 1. Initialize the MediaCapture with InitializeAsync.

m_mediaCapture = new MediaCapture();
var selectedDevice = _devices.FirstOrDefault(x => x.EnclosureLocation != null && x.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Back);

await m_mediaCapture.InitializeAsync(new MediaCaptureInitializationSettings
{
     VideoDeviceId = selectedDevice.Id
});

Step 2. Select the resolution and SubType of the preview and record streams.

var encodingPreviewProperties = m_mediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(MediaStreamType.VideoPreview);
var encodingRecorderProperties = m_mediaCapture.VideoDeviceController.GetAvailableMediaStreamProperties(MediaStreamType.VideoRecord);
 
var selectedPreviewProperties = encodingPreviewProperties.First(x => ((VideoEncodingProperties)x).Width == 800);    
await _mediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoPreview, selectedPreviewProperties);
 
var selectedRecordingProperties = encodingRecorderProperties.First(x => ((VideoEncodingProperties)x).Width == encodingRecorderProperties.Max(y => ((VideoEncodingProperties)y).Width));

await m_mediaCapture.VideoDeviceController.SetMediaStreamPropertiesAsync(MediaStreamType.VideoRecord, selectedRecordingProperties);

Step 3. Add the effect. This is done by passing a string parameter that contains the namespace and class of the VideoEffect to the AddEffectAsync method on the MediaCapture instance. The actual effects that should be used are passed within a PropertySet class as the third argument of AddEffectAsync. To the PropertySet instance, just add a KeyValuePair with "ImageProviders" as the key and a IImageProvider that contains the effect you wish to apply. If you have an effect chain, you will need to set the IImageConsumer of the effect chain that is the first effect in the chain. The first effect will have its source connected automatically to the input video within the component. The last effect will be effect rendered to the output video.

var grayscaleEffect = new GrayscaleEffect();
var brightnessEffect = new BrightnessEffect(grayscaleEffect);
var m_configurationPropertySet = new PropertySet();
configuration = new VideoEffectConfiguration((IImageConsumer)grayscaleEffect, brightnessEffect);           

m_configurationPropertySet.Add(new KeyValuePair<string, object>("Configuration", configuration)); 

Finally, a call to StartPreviewAsync() will begin the camera preview.

await _mediaCapture.StartPreviewAsync();

Recording

It is not necessary to stop the Preview when Recording is started. However, the effect MUST be applied to only one of these streams; otherwise, the effect will run TWICE within both the preview output and the recording.

First, call ClearEffectsAsync and pass the MediaStreamType.VideoPreview as an argument to remove the effect from the preview. Then, add the effect to the VideoRecord stream using the same effect parameters before called StartRecordToStorageFileAsync (or any other recording method).

await m_mediaCapture.ClearEffectsAsync(MediaStreamType.VideoPreview);
await m_mediaCapture.AddVideoEffect("Lumia.Imaging.VideoEffect", false, propertySet);
await m_mediaCapture.StartRecordToStorageFileAsync(MediaEncodingProfile.CreateMp4(VideoEncodingQuality.HD720p), _tempStorageFile);
Show: