Digital Media
Add Video To Controls And 3D Surfaces With WPF
Lee Brimelow
This article discusses:
|
This article uses the following technologies: .NET Framework 3.0 |
Contents
Technical Overview
Using the MediaElement Control
Using the MediaPlayer Class
Embedding Video inside WPF Controls
The Wet Floor Technique
Mapping Video onto 3D Surfaces
Rendering Bitmaps from Video Frames
Additional Audio Capabilities in WPF
Summary
When the buzz surrounding Windows Presentation Foundation (WPF) began circulating throughout the Adobe Flash community, my initial reaction was one of skepticism. That Microsoft had introduced a competing technology led many Flash developers, including myself, to lash out against the company in support of our beloved platform. Then I received an e-mail message saying that Microsoft was sending a technical evangelist to the frog design studio in San Francisco to demo Windows® Presentation Foundation, and I saw it as a perfect opportunity to show everyone why Flash was superior in every way.
Halfway through Karsten Januszewski's presentation, he showed off the now-famous North Face demo (channel9.msdn.com/Showpost.aspx?postid=116327), which features a 3D carousel in which curved 3D mesh objects are mapped with high-quality Windows Media® Video (WMV) clips. The demo was created jointly by Fluid, an interactive studio also located in San Francisco, and members of the Microsoft Windows Presentation Foundation team. My prejudices were obliterated as reality set in. This type of presentation would be difficult or impossible to accomplish with Flash-or any other platform for that matter.
Not long after that session, I swallowed my pride and downloaded the Microsoft® .NET Framework 3.0 runtime, thus beginning my career as a Windows Presentation Foundation interactive designer. Since that time, the whole Flash versus Windows Presentation Foundation debate has dissipated due to the fact that there aren't many areas in which the two technologies actually compete. The debate may start raging again when Microsoft releases its latest solution for delivering cross-platform Web pages that include graphics, video, animation, and audio. This upcoming release is codenamed "WPF/E" and a preview is available on the "WPF/E" developer center at msdn2.microsoft.com/bb187358.aspx.
The ability to map video to the 3D surfaces certainly provided the necessary eye candy to grab the attention of interactive designers. But this feature only scratches the surface of what's possible when integrating audio and video into Windows Presentation Foundation. This article will get you up to speed with Windows Presentation Foundation media integration and give you the knowledge you need to add media files and that optional visual interest to your own applications.
Technical Overview
The two most common ways of adding digital media files to Win32® applications today are by using the DirectShow® API or by embedding the Windows Media Player ActiveX® control. Using DirectShow gives developers ultimate control over media integration, but is much more complex than simply embedding the Windows Media Player ActiveX control.
Using DirectShow is still a viable option for Windows Presentation Foundation applications that need fine-grained control over media display and management. You might also implement DirectShow in a Windows Presentation Foundation application when an existing Win32 app is being ported to Windows Presentation Foundation and the .NET Framework 3.0.
Playing media using the Windows Media Player ActiveX control is still an option in Windows Presentation Foundation, but it makes much more sense to use the new MediaElement control. It allows you to do essentially the same thing, as you'll see in a few moments; however, it provides better integration with Windows Presentation Foundation. Specifically, it gives you the ability to composite media with alpha-blending, and it also lets you render onto 3D surfaces.
Audio and video integration with Windows Presentation Foundation gives developers multiple options to choose from, each with its own levels of complexity and control over media files. Regardless of the control or class used, what happens behind the scenes remains pretty much the same. The general idea is that when your application is told to play a media file in Windows Presentation Foundation, it launches the Windows Media Player runtime in the background to play the file. The fact that Windows Presentation Foundation uses Windows Media Player and does not contain its own media rendering engine presents some advantages and disadvantages. Piggybacking on Windows Media Player is somewhat analogous to using the ActiveX control.
Since Windows Media Player is responsible for Windows Presentation Foundation media playback, it's easy to talk about which media formats are compatible with Windows Presentation Foundation. Basically, any formats an end user can play through an existing Windows Media Player installation are the formats available to Windows Presentation Foundation applications. In order to add support for an additional media type, the user would have to install a new video codec.
Be aware that since the Windows Media Player runtime has to launch to play media files, there will be a delay in the media's initial playback as the runtime is loaded in the background. This can make precise control over media difficult, especially when trying to do things like syncing audio or video to animations or other events in Windows Presentation Foundation. One way to get around this is by preloading media files and keeping them in a paused state until you are ready to use them. Another options is that by using slip in the animation engine, you can make other animations dependent on the MediaElement, so at least everything will delay equally.This is not an optimal workflow and the Windows Presentation Foundation media team is working on a solution for the next version. Luckily, for audio playback there's an alternative-the System.Media.SoundPlayer class that allows for simple playback of WAV files without the need for Windows Media Player. We'll look at using this class later in this article.
Using the MediaElement Control
By far the easiest way to add media to a Windows Presentation Foundation application is by using the new MediaElement control. This control, like all others in Windows Presentation Foundation, can be added using either XAML or in the codebehind file using either C# or Visual Basic®. MediaElement enables basic playback and control over media files and can be run in one of two modes: Independent or Clock.
Independent mode, which is the default, works most closely like a traditional media player:
<MediaElement Source="myMedia.wmv" Width="320"
Height="240" LoadedBehavior="Manual" />
You set the Source property, which will be a URI to a media file. What happens when the media file is loaded depends on the LoadedBehavior property. This property must be set to a value from the MediaState enumeration: Close, Manual, Pause, Play, or Stop. In order to control media playback using the Play, Pause, or Stop methods of the MediaElement control, the LoadedBehavior property must be set to Manual.
Figure 1 shows an example of a simple media player using the MediaElement control running in Independent mode. Figure 2 shows the XAML and C# code used to implement this control.
Figure 2 Implementing the MediaElement Control
XAML
XAML
<Window x:Class="VideoProject.Window1"
xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
Title="VideoProject" Height="800" Width="600">
<Grid>
<StackPanel HorizontalAlignment="Center" VerticalAlignment="Center">
<MediaElement Name="myMedia" Source="myMedia.wmv"
LoadedBehavior="Manual" Width="320" Height="240" />
<StackPanel Orientation="Horizontal" Margin="0,10,0,0">
<Button Content="Play" Margin="0,0,10,0"
Padding="5" Click="mediaPlay" />
<Button Content="Pause" Margin="0,0,10,0"
Padding="5" Click="mediaPause" />
<Button x:Name="muteButt" Content="Mute"
Padding="5" Click="mediaMute" />
</StackPanel>
</StackPanel>
</Grid>
</Window>
C#
C#
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Shapes;
namespace VideoProject
{
public partial class Window1 : System.Windows.Window
{
public Window1()
{
InitializeComponent();
myMedia.Volume = 100;
myMedia.Play();
}
void mediaPlay(Object sender, EventArgs e)
{
myMedia.Play();
}
void mediaPause(Object sender, EventArgs e)
{
myMedia.Pause();
}
void mediaMute(Object sender, EventArgs e)
{
if (myMedia.Volume == 100)
{
myMedia.Volume = 0;
muteButt.Content = "Listen";
}
else
{
myMedia.Volume = 100;
muteButt.Content = "Mute";
}
}
}
}
Figure 1** A Simple MediaElement Control **
Running the MediaElement control in Clock mode enables you to control media playback using the robust Windows Presentation Foundation animation engine. Clock mode can be enabled either by targeting the MediaElement as part of a MediaTimeline or by explicitly creating a MediaClock from a MediaTimeline and assigning it to the MediaElement (though that's not a recommended approach). This may sound confusing, but essentially it just means you can attach the MediaElement to a Storyboard and animate its position using the Windows Presentation Foundation animation engine. This allows you to sync up media playback to other animations in your application.
Here's an example of the XAML code needed to playback media in Clock mode using a Storyboard:
<MediaElement Name="myMedia" Width="320" Height="240" />
<Grid.Triggers>
<EventTrigger RoutedEvent="Grid.Loaded">
<EventTrigger.Actions>
<BeginStoryboard>
<Storyboard Storyboard.TargetName="myMedia">
<MediaTimeline Source="myMedia.wmv"
BeginTime="00:00:00" Duration="00:05:00" />
</Storyboard>
</BeginStoryboard>
</EventTrigger.Actions>
</EventTrigger>
</Grid.Triggers>
Note that the source of the media file is set in MediaTimeline rather than in MediaElement. MediaTimeline targets the MediaElement control via the Storyboard.TargetName attached property. Clock mode can be confusing if you're not familiar with the Windows Presentation Foundation animation engine, so make sure you're comfortable with basic Windows Presentation Foundation animation before venturing down this path.
After you've set up MediaElement and configured its mode, you can then read and write to a whole slew of properties to further enhance the media playback experience. To get or set the position of the media file, you can simply set the Position property with a valid TimeSpan object. You can display the buffering and playback progress to the user by reading the BufferingProgress and DownloadProgress properties. For audio control, the Balance and Volume properties can be set to customize or animate a media's audio track. Now think about if these values were databound to a couple of sliders-you could give the user complete control over the media's aural experience!
MediaElement also fires a number of useful events. MediaOpened and MediaEnded are handy for triggering other parts of your application based on whether the media has started or ended. The BufferingStarted and BufferingEnded events allow you to display feedback to the user when the video freezes during a buffering operation. If used in conjunction with the BufferingProgress property, they let you display a buffering percentage to the user similar to what is seen when viewing streaming video with Windows Media Player.
Based on my own experience-and reinforced by conversations with the Windows Presentation Foundation media team-I recommend you normally use the MediaElement control in Independent mode unless Clock mode is absolutely necessary. For standard media playback and control, there's no need to deal with Storyboards or MediaTimelines. And MediaElement should be your default choice for media integration since the ability to add it using XAML lets designers implement media, leaving the developer free to do other things.
Using the MediaPlayer Class
If the MediaElement control doesn't provide enough functionality and control over playback for your application, the next option for you to consider is using the MediaPlayer class located in the System.Windows.Media namespace. Since this is not a Windows Presentation Foundation control element, the only way to implement a MediaPlayer is to create it in the codebehind file using either C# or Visual Basic. Here's a simple example that shows how to implement a MediaPlayer using C#:
MediaPlayer mp = new MediaPlayer();
Try {
mp.Open(new Uri("myMedia.wmv"));
}
catch(Exception ex) {
MessageBox.Show(ex.Message);
}
The properties and methods available in the MediaPlayer class are strikingly similar to those of the MediaElement control. This is not merely a coincidence-the MediaElement is essentially a high-level wrapper of the MediaPlayer class. The major caveat with using MediaPlayer is that it doesn't have any direct visual representation and can't be added directly to the visual tree. To play video you need to draw the MediaPlayer onto a visual surface and then add that surface to the visual tree. This makes MediaPlayer a good choice for playing audio as, obviously, no visual representation is needed. For straight video playback, the MediaElement control is a much better option since it inherits from the UIElement class and thus can be directly added to the visual tree in your applications.
Just like the MediaElement control, you can run a MediaPlayer instance in either Independent or Clock mode. As for playback control, again this is essentially a carbon copy of the MediaElement control. So what's different about the MediaPlayer? Well, for one, there are no LoadedBehavior or UnloadedBehavior properties to deal with, as these are unique to the MediaElement control. If you are planning to draw media to 3D surfaces using the VideoDrawing class, then you must use MediaPlayer because the VideoDrawing's Player property won't accept a MediaElement control as its value. (Don't worry-I'll show you how to draw video to 3D and other surfaces a little later.)
Another advantage (or disadvantage, depending on your situation) is that since the MediaPlayer is not implemented via XAML, it doesn't become part of the visual tree. Not being a part of the visual tree can be beneficial if your media file doesn't need visual representation. There are also some differences in how you load a media file into your application. In the MediaElement control you set its Source property, but when using the MediaPlayer class you call the Open method and pass in a valid Uri object.
So if your application doesn't require media files to be implemented using XAML or to have an automatic visual representation, the Media Player is probably your best option since it gives you all the capabilities of the MediaElement control without the dreaded additional overhead.
Embedding Video inside WPF Controls
Since the MediaElement control inherits from the System.Windows.UIElement class, it can be placed anywhere any other UIElement control can be. One such example would be nesting a MediaElement control inside a Button control, thus creating a video button. This can be accomplished using this XAML code:
<Button Name="myVideoButton" Click="playVideo"
Width="320" Height="240">
<MediaElement Name="myVid" Source="myMedia.wmv"
LoadedBehavior="Manual" Width="320"
Height="240" />
</Button>
In this code I'm embedding MediaElement inside of a Button and setting its LoadedBehavior to Manual. The click event of the Button calls a method (not shown) that tells the MediaElement to begin playing. With a few lines of code you've created an interactive video thumbnail. All kinds of creative possibilities open up due to the fact that Windows Presentation Foundation treats the MediaElement control just like any other UI element in the visual tree. As you'll see in the next section, you can paint the video playing in the MediaElement onto other Windows Presentation Foundation objects to create incredibly rich user experiences.
The Wet Floor Technique
In recent times, glass and reflections have taken over the interfaces of everything in sight. Without a doubt, someday you will be asked to deliver such a technique in Windows Presentation Foundation, and fortunately it's easy. The extensive brush system in Windows Presentation Foundation allows you paint one control onto another with ease. In this example, I'll paint the video playing in a MediaElement control onto a Rectangle using a VisualBrush, creating a perfect reflection of the video. In the design world, this effect is known as a wet floor.
The easiest way to create a wet floor effect is by placing your MediaElement control in the first slot of a StackPanel control. Below that you then create a Rectangle with the same dimensions as the video. Next, use a VisualBrush as the fill for the Rectangle and databind its Visual property to the MediaElement control. Finally, flip the reflection to create a mirror image and apply an OpacityMask to fade out the reflection.
Figure 3 shows the completed effect, while Figure 4 shows the XAML code. This type of visual manipulation would have been incredibly complicated in traditional Win32 programming. And, thanks to the powerful databinding features in Windows Presentation Foundation, this is a live reflection that automatically updates whenever your video does.
Figure 4 XAML Used to Create the Wet Floor
<Grid x:Name="myGrid">
<StackPanel VerticalAlignment="Center">
<MediaElement Name="myVid" Source="myMedia.wmv"
LoadedBehavior="Play" Width="320" Height="240" />
<Rectangle Width="320" Height="240">
<Rectangle.Fill>
<VisualBrush Visual="{Binding ElementName=myVid}" />
</Rectangle.Fill>
<Rectangle.OpacityMask>
<LinearGradientBrush StartPoint="0.5,0" EndPoint="0.5,1">
<GradientStop Color="#AA000000" Offset="1" />
<GradientStop Color="#00000000" Offset="0" />
</LinearGradientBrush>
</Rectangle.OpacityMask>
<Rectangle.RenderTransform>
<TransformGroup>
<ScaleTransform ScaleY="-1" />
<TranslateTransform Y="242" />
</TransformGroup>
</Rectangle.RenderTransform>
</Rectangle>
</StackPanel>
</Grid>
Figure 3** The Completed Wet Floor **
Mapping Video onto 3D Surfaces
If I had to pick the one area that seems to have created the biggest buzz around Windows Presentation Foundation, it would be the ability to map video files onto 3D mesh objects. The North Face demo showed just how powerful this could be in helping to create a visually stunning user interface. While many people have written off this effect as either too gimmicky or too resource-intensive, I feel that, when done in moderation, it can turn an ordinary application into one that provides an outstanding experience for users.
As for performance, of course there's quite a bit of overhead involved with displaying and animating 3D meshes. If you add to that displaying full-motion video that is being mapped onto the 3D meshes, you are talking about a ton of overhead. Just as with drinking mocha frappucinos, the key here is moderation. Overall, I've found the 3D performance in Windows Presentation Foundation excellent, so long as the complexity (polygon count) of the 3D meshes is kept to the minimum necessary. The Windows Presentation Foundation media team is committed to reducing the expense of 3D media in future versions, although it will always be more expensive than 2D media.
Before looking at how to map video onto 3D objects, let's first review what it takes to set up a 3D scene in Windows Presentation Foundation. First and foremost, you need a 3D object to place into the scene. There are no 3D primitive classes that are a part of the .NET Framework 3.0. But someone at Microsoft created a class called Mesh3DObjects that can create a wide range of primitive objects.
There are certain sets of elements required in order to create a 3D scene in Windows Presentation Foundation. First, you need the 3D object, and this object needs to have some kind of material in order to be viewable in the scene. Materials are created using brushes like SolidColorBrush, LinearGradientBrush, ImageBrush, or a VisualBrush. You'll see in a few moments how a VisualBrush can be used to map video, or any other visual element for that matter, onto a 3D surface. The technique is essentially the same as the wet floor technique that I discussed earlier.
The 3D mesh and materials need to be grouped inside of a GeometryModel3D element, and doing so creates a valid Windows Presentation Foundation 3D object. For a 3D scene to be visible, you'll need to add one or more lights to the scene. Then, just as a photograph isn't possible without a camera, a 3D scene in Windows Presentation Foundation isn't either. Finally, to complete the scene, all of these items need to be wrapped in a Viewport3D control in order to seal the deal.
Once you have a functional 3D scene in place, you can start having fun mapping video and other media to the 3D meshes. This is accomplished using a VisualBrush to draw the video onto the 3D surface. In the following code snippet, I am applying a VisualBrush to a 3D object's DiffuseMaterial:
<GeometryModel3D.Material>
<DiffuseMaterial>
<DiffuseMaterial.Brush>
<VisualBrush>
<VisualBrush.Visual>
<MediaElement Source="myMedia.wmv" />
</VisualBrush.Visual>
</VisualBrush>
</DiffuseMaterial.Brush>
</DiffuseMaterial>
</GeometryModel3D.Material>
Figure 5 shows a single frame of the video mapped onto the Windows Presentation Foundation 3D surface.
Figure 5** Mapped to a WPF 3D Surface **
There's another alternative you can use to map video onto 3D surfaces in Windows Presentation Foundation. The VideoDrawing class, which is a part of the System.Windows.Media namespace, does precisely what its name implies. It can draw video coming from a MediaPlayer instance via a DrawingBrush onto any brush-compatible surface. In order to achieve this, you need to set the Drawing property of the DrawingBrush to an instance of the VideoDrawing class. This VideoDrawing instance can render video by setting its Player property to a MediaPlayer instance containing a video file. Finally, the DrawingBrush can be applied to a 3D object's DiffuseMaterial's Brush property. The following code snippet shows an example implementation using C#:
MediaPlayer mp = new MediaPlayer();
mp.Open(myValidMediaUri);
VideoDrawing vd = new VideoDrawing();
vd.Player = mp;
DrawingBrush db = new DrawingBrush();
db.Drawing = vd;
diffuseMat.Brush = db;
vd.Player.Play();
One of the major differences between using a VisualBrush or the VisualDrawing class is that a VisualDrawing must be created and implemented in the codebehind file (using C# or Visual Basic). Furthermore, the VideoDrawing class will only work with MediaPlayer instances and not with the MediaElement control. When it comes to performance, there isn't much difference between the two methods since both require Windows Presentation Foundation to create an intermediary render surface in order to correctly map the video to 3D.
Rendering Bitmaps from Video Frames
A video application may need to essentially take a screen grab of the video stream and render it to a bitmap. One such example is seen in some commercial DVD playback software where the user can click on a camera icon at any time during playback to take a screen grab from the video. Another use could be for creating thumbnails of a set of video files if you didn't want all of them actually connected to MediaElement controls. There is also some potential for creating runtime video effects, although for this type of thing you'd be much better served by diving into the DirectShow API.
Like most things in Windows Presentation Foundation, there are multiple ways of accomplishing a task, and rendering bitmaps of video frames is no different. No matter what method is used, the core functionality for rendering bitmaps of visual elements in Windows Presentation Foundation is handled by the RenderTargetBitmap class, which is a part of the Windows.Media.Imaging namespace. This amazing class can create bitmaps from any item in the visual tree of a Windows Presentation Foundation application.
To render a bitmap using this class, first you need to create a new DrawingVisual object, which gets passed to the RenderTargetBitmap.Render method. Next you create a new DrawingContext object that you'll pass to the DrawingVisual.Render method. With this DrawingContext instance, you can draw all kinds of graphics into the DrawingVisual object. To draw a frame of video, you can call the DrawingContext.DrawVideo method and pass in a MediaPlayer instance that is currently playing the video file. You'll also need to pass in a Rect value that determines the area of the video file that will be captured.
At this point, you can actually render a bitmap from the DrawingVisual instance that now contains your captured video frame. To do this, you just need to pass it to the RenderTargetBitmap.Render method. Now your RenderTargetBitmap instance contains the video frame, but it won't be a bitmap until you pass it to the BitmapFrame.Create method. To see the bitmap frame, you can simply set this method call as the value of an Image control's Source property. The following snippet shows a C# implementation of this frame-capture technique:
RenderTargetBitmap rtb = new RenderTargetBitmap(320, 240, 1 / 200,
1 / 200, PixelFormats.Pbgra32);
DrawingVisual dv = new DrawingVisual();
DrawingContext dc = dv.RenderOpen();
dc.DrawVideo(myMediaPlayer, new Rect(0, 0, 320, 240));
dc.Close();
rtb.Render(dv);
Image im = new Image();
im.Source = BitmapFrame.Create(rtb);
The target bitmap's format is set by passing a valid PixelFormats enumeration value to the RenderTargetBitmap constructor. The ability to render bitmaps from video frames may at first seem like a niche feature, but there are actually many uses for this type of functionality beyond the simple video screen capture tool.
Additional Audio Capabilities in WPF
Most everything I've discussed so far about Windows Presentation Foundation media integration holds true for both video and audio. There are, however, several additional classes available for handling audio playback in Windows Presentation Foundation. As I mentioned earlier, media played in Windows Presentation Foundation is actually driven by a set of Windows Media Player runtimes that need to be loaded before the file can be played back. This can cause major issues when you're trying to synchronize media files to user events or animations, though it can be mitigated with slip behavior in timelines. When it comes to audio, there are some alternatives that can bypass this problem.
The SoundPlayer class, which is a part of the System.Media namespace, provides a simple interface for loading and playing back audio WAV files. These WAV files can be external, streaming, or embedded resources. This method of audio playback has much lower overhead than playing an audio file through a MediaElement control since it does not at all depend on Windows Media Player for playback.
To load a WAV file into a SoundPlayer instance, you first need to set the SoundLocation property to the location of the WAV file and then call the Load method to start the loading process. Once the file is fully loaded you can play it using the SoundPlayer.Play method. The file won't start playing until it is fully loaded into memory.
This lightweight audio playback is also available in XAML using the SoundPlayerAction control, which lets you play a WAV file inside of an EventTrigger. This is very useful for things like interface sounds for button rollovers, for example. To use it, you simply place it into the EventTrigger.Actions block of the EventTrigger. Following is a code snippet of SoundPlayerAction being called in response to a button's MouseEnter event:
<EventTrigger RoutedEvent="Button.MouseEnter" SourceName="myButton">
<EventTrigger.Actions>
<SoundPlayerAction Source="media/overSound.wav"/>
</EventTrigger.Actions>
</EventTrigger>
Summary
I've explored here the available methods for integrating audio and video into Windows Presentation Foundation applications. If Windows Presentation Foundation does not provide the level of control needed for your project, you can always revert to using the DirectShow API for media integration. But the ability to add audio and video using XAML allows designers to handle simple media integration so that developers can focus on more complex coding. I am no longer the sceptic I once was. The amazing visual richness available in Windows Presentation Foundation makes media integration not only possible, but incredibly exciting!
Lee Brimelow, a Senior Design Technologist at frog design, is an award-winning interactive designer who specializes in Flash and WPF. He also runs the popular WPF technology blog at www.thewpfblog.com.