Chroma Key Demo for Windows Phone 8.0

Chroma Key Demo is a Lumia Developer example demonstrating the Lumia Imaging SDK chroma key filter. Tap on the upper original viewfinder to select a chroma key color (for example, a green backdrop canvas) and use the color distance slider to adjust the threshold of the color range to be made transparent. Tap on the lower modified viewfinder to toggle between a video background and a "transparent" background.

Compatibility

• Compatible with Windows Phone 8.0.
• Tested with Nokia Lumia 1020 and Nokia Lumia 1520.
• Developed with Visual Studio 2012 Express for Windows Phone 8.
• Compiling the project requires Lumia Imaging SDK.

Design

The application is a single page app with two viewfinders, both of which serve two purposes. The viewfinder in the top of the screen shows an unmodified live camera preview feed and tapping anywhere on it picks the color in the tapping point from the feed and that color is used as a chroma key color. The viewfinder in the bottom of the screen shows a chroma key filtered version of the camera preview feed, meaning that the selected color is converted into a transparent area, and thus a background behind the viewfinder can shine through. The application has two different backgrounds for this purpose, a static "transparent" looking background familiar from graphics editing applications, and an underwater trip video. You can toggle between the transparency and the video by tapping anywhere on the lower viewfinder.

At the bottom of the application page there is a rectangular indicator()** for the currently selected chroma key color and a slider to control the chroma key effect color distance.

*Figure 1. White color made transparent and underwater trip video shown in the background*

*Figure 2. White color made transparent and transparency checker board shown in the background*

Architecture overview

The application uses PhotoCaptureDevice for the camera preview feed and the upper viewfinder in the UI is implemented with a
VideoBrush rendering on a Canvas. The modified lower viewfinder is implemented simply as an Image that displays a WriteableBitmap that is rendered using the Lumia Imaging SDK and using its ChromaKeyFilter. The "transparent" looking background image and the background video are not explicitly blended together, but they are instead simply layered under the now semitransparent viewfinder image in XAML.

Initializing and rendering the modified camera preview

Here's how the rendering pipeline is initialized in this application. PhotoCaptureDevice is initialized in App when the application is launched, and it is referenced from the MainPage with the static App.Camera property. ViewfinderBrush is the VideoBrush rendering the upper unmodified viewfinder, FilteredImage displays the modified lower viewfinder and ColorBorder is the rectangular selected color indicator.

// ...

public partial class MainPage : PhoneApplicationPage
{
    private WriteableBitmap _bitmap = null;
    private DispatcherTimer _timer = null;
    private Color _color = new Color();
    private bool _initialized = false;
    private bool _rendering = false;
    private CameraPreviewImageSource _source = null;
    private WriteableBitmapRenderer _renderer = null;
    private FilterEffect _effect = null;
    private IList<IFilter> _filters = null;
    private ChromaKeyFilter _chromaKeyFilter = null;
    private RotationFilter _rotationFilter = null;

    ...

    private void Initialize()
    {
        // Initialize camera

        var rotation = App.Camera.SensorLocation == CameraSensorLocation.Back ?
            App.Camera.SensorRotationInDegrees : - App.Camera.SensorRotationInDegrees;

        ViewfinderBrush.SetSource(App.Camera);
        ViewfinderBrushTransform.Rotation = rotation;

        // Setup image processing pipeline

        _source = new CameraPreviewImageSource(App.Camera);
        _rotationFilter = new RotationFilter(rotation);
        _chromaKeyFilter = new ChromaKeyFilter();

        _filters = new List<IFilter>();
        _filters.Add(_rotationFilter);
        _filters.Add(_chromaKeyFilter);

        _effect = new FilterEffect(_source);
        _effect.Filters = _filters;

        _bitmap = new WriteableBitmap((int)App.Camera.PreviewResolution.Width, (int)App.Camera.PreviewResolution.Height);
        _renderer = new WriteableBitmapRenderer(_effect, _bitmap, OutputOption.Stretch);

        FilteredImage.Source = _bitmap;

        _color = Color.FromArgb(0xFF, 0xFF, 0xFF, 0xFF);

        ColorBorder.Background = new SolidColorBrush(_color);

        // Start rendering timer

        _timer = new DispatcherTimer { Interval = TimeSpan.FromMilliseconds(10) };
        _timer.Tick += DispatcherTimer_Tick;
        _timer.Start();

        _initialized = true;
    }

    private async void DispatcherTimer_Tick(object sender, System.EventArgs e)
    {
        if (_initialized && !_rendering)
        {
            try
            {
                _chromaKeyFilter.Color = Windows.UI.Color.FromArgb(_color.A, _color.R, _color.G, _color.B);
                _chromaKeyFilter.ColorDistance = DistanceSlider.Value;

                await _renderer.RenderAsync();

                _bitmap.Invalidate();
            }
            catch (Exception)
            {
            }

            _rendering = false;
        }
    }

    ...
}

Playing video behind the modified camera preview feed

The structure of the lower camera preview feed is following. The "transparency" background is a bitmap set as the background for the whole viewfinder and on top of that is a grid in which there is a MediaElement playing the video and an Image displaying the modified camera preview feed. FilteredImage is drawn on top of the MediaElement because it is introduced later in the XAML declaration.

<Border BorderBrush="{StaticResource PhoneSubtleBrush}" Grid.Row="1" BorderThickness="2">
    <Border.Background>
        <ImageBrush ImageSource="/Assets/Graphics/Background.png" Stretch="UniformToFill"/>
    </Border.Background>

    <Grid >
        <MediaElement x:Name="BackgroundMediaElement" Stretch="Fill" Volume="0"
                      Source="/Assets/Video/oceantrip-small.mp4"
                      MediaEnded="BackgroundMediaElement_MediaEnded"
                      Visibility="Visible" AutoPlay="True"/>

        <Image x:Name="FilteredImage" RenderTransformOrigin="0.5,0.5" Stretch="Fill" Tap="FilteredImage_Tap">
            <Image.RenderTransform>
                <CompositeTransform x:Name="FilteredImageTransform"/>
            </Image.RenderTransform>
        </Image>
    </Grid>
</Border>

Picking the chroma key color

When user taps on the upper unmodified viewfinder, the application selects a new chroma key color from the tap point in the camera preview feed. Picking the color is implemented so that a single frame is rendered from the PhotoCaptureDevice and the tap point pixel data is then read from the rendered bitmap.

// ...

public partial class MainPage : PhoneApplicationPage
{
   // ...

    private async void ViewfinderCanvas_Tap(object sender, System.Windows.Input.GestureEventArgs e)
    {
        var point = e.GetPosition(ViewfinderCanvas);
        var bitmap = new WriteableBitmap((int)ViewfinderCanvas.ActualWidth, (int)ViewfinderCanvas.ActualHeight);

        using (var source = new CameraPreviewImageSource(App.Camera))
        using (var effect = new FilterEffect(source))
        using (var renderer = new WriteableBitmapRenderer(effect, bitmap, OutputOption.Stretch))
        {
            effect.Filters = new List<IFilter>()
            {
                new RotationFilter(_rotationFilter.RotationAngle)
            };

            await renderer.RenderAsync();

            var picked = bitmap.Pixels[((int)point.Y) * bitmap.PixelWidth + ((int)point.X)];

            _color = new Color
            {
                A = 0xFF,
                R = (byte)((picked & 0x00FF0000) >> 16),
                G = (byte)((picked & 0x0000FF00) >> 8),
                B = (byte)(picked & 0x000000FF)
            };
        }

        ColorBorder.Background = new SolidColorBrush(_color);
    }

    // ...
}

See also

• ChromaKeyFilter

Downloads

This example application is hosted in GitHub, where you can check the latest activities, report issues, browse source, ask questions, or even contribute to the project yourself.

Note: In order to deploy the application binary file yourself to a Windows Phone device, not via store, you must be a registered developer and your phone must be registered for development. For more details, see the deploying and running apps on a Windows Phone device (MSDN).