Introducing Some New Video Mixing Renderer Sample Applications

  • Article

by Michael Blome

     Microsoft Corporation

     April 2003

Abstract

The Video Mixing Renderer (VMR) is a Microsoft® DirectShow® filter that provides applications based on DirectShow with many new capabilities for compositing and blending multiple video streams at run time. This whitepaper discusses the sample applications for both versions of the VMR, VMR-7 and VMR-9, which are included in the Microsoft DirectX® 9.0 Software Development Kit (SDK). It highlights the SDK samples to demonstrate the various capabilities of the VMR. This paper assumes you have a basic familiarity with DirectShow application programming.

Introduction

The Video Mixing Renderer (VMR) is a Microsoft® DirectShow® filter that provides applications based on DirectShow with many new capabilities for compositing and blending multiple video streams at run time. The first version of this filter, the VMR-7, is available only on Microsoft Windows® XP Home Edition, Windows XP Professional, and the Windows Server 2003 family. A second version of the filter, the VMR-9, is available through the Microsoft DirectX® 9.0 Software Development Kit (SDK) and is supported on all the same operating systems as DirectX. For more information about the VMR-9 hardware requirements, see the DirectShow SDK documentation. Although this whitepaper specifically discusses the sample applications for the VMR-7, most of the information presented is also relevant for the corresponding VMR-9 samples in the Microsoft DirectX® 9.0 SDK.

This article contains the following sections:

  • The VMR Compared to Earlier DirectShow Renderers
  • Capabilities of the VMR
  • VMR System Requirements
  • VMR Modes of Operation
  • The VMR Sample Applications
    • Windowless: Basic Playback and Image Capture
    • Blender: Mixing Multiple Input Streams
    • Watermark: A Logo Of Your Own
    • Text: Using the Bitmap for Subtitles
    • Ticker: Scrolling Dynamic Text on Video
    • Cube: Mixing Streams Onto a D3D Surface
    • VMRMulti: The Ultimate VMR Sample

The VMR Compared to Earlier DirectShow Renderers

With the old filters, different renderers would be required in the graph depending on the hardware configuration.

The Video Renderer filter was used to render a single video stream in non-video port scenarios. It was based on graphics hardware technology which is now over five years old, and on an older version of DirectDraw. In certain scenarios, it uses GDI for rendering. This is done either to conserve video resources, which were much more limited five years ago, or else to overcome limitations in DirectDraw that related to multi-monitor support. The VMR never uses GDI for rendering and is based completely on DirectDraw 7.

In scenarios involving either a video port or multiple video input streams, prior to the VMR the Overlay Mixer filter was used for rendering. This filter only uses the hardware overlay on the graphics card, and so is generally limited to the one overlay surface provided by most cards. The Overlay Mixer performs destination color keying, but it is not capable of true video mixing. Because it does not have a window manager, it must use a second filter, the Video Renderer, for window management. The VMR is capable of true alpha mixing, and can create multiple overlays in software in addition to the hardware overlays.

In video port scenarios where applications were overlaying closed captioning or other VBI data on the video, an additional filter, the VBI Surface Allocator, was required in order to allocate the additional video memory for the VBI text.

For ISVs, the VMR simplifies application development by combining allocation and rendering functionality into a single filter which is used in all scenarios. With the VMR, the VBI Surface Allocator is no longer needed. This filter is replaced in Windows XP Home Edition and Windows XP Professional by the new Video Port Manager filter which performs all of the video port tasks previously performed by the Overlay Mixer.

The VMR is more robust than the earlier renderers, in part because it only uses DirectDraw 7 interfaces, as opposed to the old renderers which used a mixture of interfaces from older and newer versions of DirectDraw. The VMR also employs a new image-presentation mechanism which is designed for current and future generations of adapters, which have support for D3D, increased VRAM and video memory bandwidth, and hardware acceleration features. With the VMR, the focus is on front-end processing, and reduced dependence on videoports and overlays. But even with all its new functionality, the VMR is designed for maximum compatibility with existing applications.

The VMR is also extensible. Applications can provide their own sub-components to perform custom video effects and/or take control of the allocation and rendering process.

Capabilities of the VMR

The Video Mixing Renderer (VMR) supports the following new features:

  • Real mixing of multiple video streams, using the alpha-blending capabilities of Direct3D hardware devices.
  • The ability to plug in your own compositing component to implement effects and transitions between multiple video streams entering the VMR.
  • True windowless rendering. It is no longer necessary to make the video playback window a child of the application's window in order to contain video playback. The VMR's new windowless rendering mode allows applications to easily host video playback within any window without having to forward window messages to the renderer for renderer-specific processing.
  • A new renderless playback mode where applications can supply their own allocator component to get access to the decoded video image prior to it being displayed on the screen.
  • Improved support for PCs equipped with multiple monitors.
  • Support for Microsoft's new DirectX Video Acceleration architecture.
  • Support for high-quality video playback concurrently on multiple windows.
  • Support for DirectDraw Exclusive Mode
  • 100% backward compatibility with existing applications.
  • Support for frame stepping and a reliable way to capture the current image being displayed.
  • The ability for applications to easily alpha-blend their own static image data (such as channel logos or UI components) with the video in a smooth flicker-free way.

VMR System Requirements

The VMR uses the graphics processing capabilities of the computer's display card exclusively; the VMR does not perform any blending or rendering of video using the host processor as doing so would greatly impact the frame rate and quality of the video being displayed. When taking advantage of the new features offered by the VMR, particularly blending of multiple video streams and/or application images, the overall performance obtained is very much dependent on the capabilities of the graphics card being used on the computer. Graphics cards that perform well with the VMR have the following hardware support built into them:

  • Support for YUV and "non-power of 2" Microsoft Direct3D® texture surfaces.
  • The ability to StretchBlt from YUV to RGB DirectDraw surfaces.
  • At least 16MB of video memory if multiple video streams are to be blended together. The actual amount of memory required is dependent on the image size of the video streams and resolution of the display mode being used. The VMR requires that the system monitor be set for a color depth of at least 16 bits. The VMR cannot be put into a run state if the monitor is set for 256 colors.
  • Support for an RGB overlay or the ability to blend to a YUV overlay surface.
  • Hardware accelerated video (support for DirectX Video Acceleration) decoding.
  • High pixel fill rates.

VMR Modes of Operation

The VMR has three modes of operation:

  • Windowed Mode
  • Windowless Mode
  • Renderless Mode

Windowed Mode

In windowed or default mode, the VMR emulates the interface of the old video renderer filter for complete compatibility with existing applications. The VMR provides its own window and supports the IVideoWindow and IBasicVideo interfaces. Windowed mode is not discussed in this paper or used in any samples since the VMR's default configuration behaves exactly the same as the old renderer. On Windows XP Home Edition, Windows XP Professional, and the Windows Server 2003 family, the VMR will be transparently selected by the Filter Graph Manager as the default video renderer. For more information on Windowed mode, see the DirectShow SDK documentation.

Windowless Mode

Windowless mode is a new feature of the VMR and it is anticipated that this mode will become the preferred way for applications to host video playback within their own application window. In windowless mode, the VMR unloads its window manager and therefore does not support the IBasicVideo or IVideoWindow interfaces. Instead, the application provides the playback window and uses the VMR's IVMRWindowlessControl interface to specify a destination rectangle in the application window's client area. In windowless mode, the VMR creates and configures a DirectDraw clipper object to ensure that the video rectangle is correctly clipped to the application's window and does not appear on any other windows used by other applications. The VMR does not subclass the application's window or install any system/process hooks.

Windowless mode is supported for single input streams and also for multiple streams using the mixer and compositor. In windowless mode, the sequence of events during connection and transition to the run state is as follows:

  • The upstream filter, typically a decoder, proposes a media type and the VMR determines whether to accept or reject it.
  • If the media type is accepted, the VMR calls the Allocator-Presenter to obtain a DirectDraw surface. If the surface is created successfully, the pins connect and the VMR is now ready to transition into the run state.
  • When the filter graph is run, the decoder calls GetBuffer on the allocator, and the VMR queries the Allocator-Presenter to ensure that the pixel depth, rectangle size, and other parameters on its DirectDraw surface are compatible with the incoming video. If they are compatible, then it returns the DirectDraw surface to the decoder. After the decoder has decoded into the surface, the VMR calls Receive on the Core Synchronization Unit, which performs a sanity check on the time stamps , and then IReferenceClock::AdviseTime is called. The Sync Unit blocks the Receive call until the presentation time arrives. At that point it calls Present() on the Allocator-Presenter, which "presents" the surface to the graphics card.

The following illustration shows the VMR in windowless mode with multiple input streams.

VMR in windowless mode image

Renderless Mode

In renderless playback mode, the VMR does not do its own rendering, but instead uses a custom Allocator-Presenter supplied by the application. This mode is useful for game writers, and HTML+TIME and other types of applications that do sophisticated video effects. Renderless playback mode enables applications to create and control their own DirectDraw surface, and/or to obtain access to the video bits at presentation time.

In renderless playback mode, the application:

  • Manages the playback window.
  • Allocates the DirectDraw object and the final frame buffer.
  • Notifies the rest of the playback system of the DirectDraw object being used.
  • "Presents" the frame buffer at the correct time.
  • Handles all Resolution Mode, monitor changes and "surface losses" — advising the rest of the playback system of these events.

The VMR:

  • Handles all timing related to presenting the video frame.
  • Provides quality-control information to the application and the rest of the playback system.
  • Presents a consistent interface to the upstream components of the playback system, which are not aware that the application is providing the frame buffer allocation and performing the rendering.
  • Provides any mixing of video streams that may be required prior to rendering.

The VMR Sample Applications

The following samples are divided into two series, Basic and Advanced. The Basic series shows what you can do using only the default components of the VMR. The first sample, Windowless, is a simple player that demonstrates how to configure the VMR for playback of a single video stream, and how to capture a video frame to a .bmp file. The next sample, Blender, shows how to configure the VMR for multiple input streams and how to control the rectangle position and alpha values for each input stream. The Watermark sample shows how to display and position an alpha-blended bitmap image, similar to a station logo, over a portion of the video rectangle. The Text and Ticker samples show how to use this same basic functionality to overlay and dynamically update text over the video rectangle.

The Advanced series of samples all demonstrate what is possible by supplying alternate components to perform custom rendering and mixing effects. VMRMulti shows how an application can supply its own Allocator-Presenter to perform cool 3-D effects. The last sample, Cube, shows how to provide your own Mixer-Compositor component to perform custom mixing effects.

VMR Samples Basic Series

Windowless: Basic Playback and Image Capture

Windowless is the simplest of the VMR samples. It, along with the other samples in the Basic series, are based on the familiar PlayWnd sample in the DirectShow SDK. In it the VMR is simply used to play back a single video stream. The only extra VMR-specific functionality is the ability to capture the current image to a .bmp file. Anyone who has struggled with the "GetCurrentImage" method on the old Video Renderer will appreciate this new, simple and reliable way to capture a poster frame.

Basic VMR Tasks for All Applications

Windowless demonstrates the basic tasks all applications must perform whenever they use the VMR in windowless or renderless mode.

  1. Verify that the VMR is present on the host system.
  2. Create and configure the VMR.
  3. Add the configured VMR to the filter graph before calling "RenderFile".
  4. Obtain a pointer to the VMR's IVMRWindowlessControl interface and call its three mandatory methods: SetVideoClippingWindow, RepaintVideo, and DisplayModeChanged.
  5. Position the video rectangle in the application window.
Verify That the VMR is Present

Because the VMR is only available on Windows XP Home Edition and Windows XP Professional, any application that uses it should verify that the VMR is present and useable on the current system, and handle the situation gracefully if it is  not present. The only way to reliably perform this check is by actually creating an instance of the VMR. Of course, if the user is running a non-Windows XP operating system, this attempt will fail. But even on Windows XP Home Edition and Windows XP Professional, the VMR might be present but not useable. In its startup code, the VMR checks the current pixel depth and other system settings and will return E_FAIL if it determines that it cannot run successfully on the current system. All this system probing is done very efficiently and does not have any noticeable impact on the application's startup time, as you may verify by running the samples. The code for VerifyVMR looks like this:

BOOL VerifyVMR(void)
{
    HRESULT hres;
    IBaseFilter* pBF = NULL;

    // Verify that the VMR exists on this system.
    hres = CoCreateInstance(CLSID_VideoMixingRenderer, NULL,
             CLSCTX_INPROC, IID_IBaseFilter, (void**)&pBF);

    if(SUCCEEDED(hres))
    {
        // VMR successfully created!
        // In this sample we just release it and create it again later
        // but if you are concerned about performance you could hang on 
        // to pBF and use it to add the VMR to the filter graph later on.
        pBF->Release();
        return TRUE;
    }
    else
    {
        // Put up a message box.
        ...
        return FALSE;
    }
}
Create and Configure the VMR

Because the VMR can accept any number of input streams, and can be configured in many different ways, automatic graph building with VMR is slightly more complicated than with the old renderer. When the VMR is to be used in windowless or renderless mode, the application should always co-create the filter and add it to the graph manually, then configure it for the correct number of input streams and the rendering mode. Only after it has been configured should the VMR be connected to any upstream filters. This means that to really take advantage of the VMR's new features, you cannot simply call "RenderFile" because this will cause the VMR to be created in its default configuration of one input stream using windowed rendering mode. In windowed mode there is no mixer compositor, so there can be no application-supplied bitmap and no mixing of input streams.

The following code snippets taken from the Windowless sample show the basic order of operations when using the VMR. Error checking and other lines have been omitted here for the sake of brevity. (The JIF macro (Jump If Failed) is defined by the application and expands to a simple HRESULT handler.)

HRESULT PlayMovieInWindow(LPTSTR szFile)
{
    ...
    // Create Filter Graph Manager using global IGraphBuilder pointer.
    JIF(CoCreateInstance(CLSID_FilterGraph, NULL, CLSCTX_INPROC_SERVER, 
                         IID_IGraphBuilder, (void **)&pGB));
    // Create and configure VMR.
    JIF(InitializeWindowlessVMR());

    // Build the graph.
    JIF(pGB->RenderFile(wFile, NULL));

    // QueryInterface for DirectShow interfaces as usual.
    JIF(pGB->QueryInterface(IID_IMediaControl, (void **)&pMC));
    JIF(pGB->QueryInterface(IID_IMediaEventEx, (void **)&pME));
    ...
}

The InitializeWindowlessVMR function does three things. It creates the VMR, adds it to the graph, and configures it. The configuration methods shown here, SetRenderingMode and SetVideoClippingWindow, are mandatory. Other optional methods on this interface are described in the DirectShow SDK documentation. The function (with error checking and cleanup code omitted) looks like this:

HRESULT InitializeWindowlessVMR()
{
    IBaseFilter* pVmr = NULL;
    
    // Create the VMR.
    HRESULT hr = CoCreateInstance(CLSID_VideoMixingRenderer, NULL,
                     CLSCTX_INPROC, IID_IBaseFilter, (void**)&pVmr);
    // Add the VMR to the filter graph.
    hr = pGB->AddFilter(pVmr, L"Video Mixing Renderer");

    // Set the rendering mode.  
    IVMRFilterConfig* pConfig;
    hr = pVmr->QueryInterface(IID_IVMRFilterConfig,
                             (void**)&pConfig);
    pConfig->SetRenderingMode(VMRMode_Windowless);
    pConfig->Release();
   
    // Tell VMR which window it should clip to.
    //  Notice we don't release pWC as we use it
    //  for RepaintVideo and DisplayModeChanged
    //  throughout the lifetime of the app.
    hr = pVmr->QueryInterface(IID_IVMRWindowlessControl,
                              (void**)&pWC);
    hr = pWC->SetVideoClippingWindow(ghApp);
    hr = pVmr->Release();

    return hr;
}

Note that we do not need to configure the VMR for the number of input streams to handle, since it defaults to one, and that is what we want for this sample.

Add the VMR to the Filter Graph

This step is performed in the InitializeWindowlessVMR function shown previously, in the AddFilter method which is called after the VMR is created and before it is configured. AddFilter may be called at any point before the graph is constructed.

Obtain a Pointer to IVMRWindowlessControl

There are three methods on the IVMRWindowlessControl interface that an application is obligated to handle. The first is SetVideoClippingWindow, which we called in the InitializeWindowlessVMR function shown previously. The other two methods are RepaintVideo and DisplayModeChanged. RepaintVideo must be called inside the application's handler for the WM_PAINT message, and DisplayModeChanged inside the application's handler for the WM_DISPLAYCHANGE message. Calling these methods is necessary to ensure that the VMR correctly updates the clipping area for your application's window when it is moved or resized. In the main Window procedure for the Windowless sample, the calls look like this:

case WM_PAINT:
    hdc = BeginPaint(hWnd, &ps); 
    if(pWC) 
       pWC->RepaintVideo(hWnd, hdc);  
    EndPaint(hWnd, &ps); 
    break;
case WM_DISPLAYCHANGE:
    if (pWC)
        pWC->DisplayModeChanged();
    break;

Position the Video Rectangle in the Application Window

Since the application has total control of its own window, it is responsible for instructing the VMR where to place the video rectangle in the window's client area. In the Windowless sample, this all happens in the InitVideoWindow function. Here the client area is sized to be identical to the native video size, while allowing for zooming of 50% and 200%. This is identical to the technique used with the old DirectShow video renderer, except for the calls to IVMRWindowlessControl::GetNativeVideoSize and IVMRWindowlessControl::SetVideoPosition. Here is the code listing for InitVideoWindow:

HRESULT InitVideoWindow(int nMultiplier, int nDivider)
{
    LONG lHeight, lWidth;
    HRESULT hr = S_OK;

    if (!pWC)
        return S_OK;

    // Read the default video size.
    hr = pWC->GetNativeVideoSize(&lWidth, &lHeight, NULL, NULL);
    if (hr == E_NOINTERFACE)
        return S_OK;

    EnablePlaybackMenu(TRUE);

    // Account for requests of normal, half, or double size.
    lWidth  = lWidth  * nMultiplier / nDivider;
    lHeight = lHeight * nMultiplier / nDivider;

    int nTitleHeight  = GetSystemMetrics(SM_CYCAPTION);
    int nBorderWidth  = GetSystemMetrics(SM_CXBORDER);
    int nBorderHeight = GetSystemMetrics(SM_CYBORDER);

    // Account for size of title bar and borders for exact match
    // of window client area to default video size.
    SetWindowPos(ghApp, NULL, 0, 0, lWidth + 2*nBorderWidth,
            lHeight + nTitleHeight + 2*nBorderHeight,
            SWP_NOMOVE | SWP_NOOWNERZORDER);

    GetClientRect(ghApp, &g_rcDest);
    hr = pWC->SetVideoPosition(NULL, &g_rcDest);

    return hr;
}

Capture the Current Image

Now that we've done the basic work, let's add one feature to our sample to make it something more than just an ordinary video playback application. The VMR provides a simple way to capture the bitmap bits of the current frame, which an application can then save to a file or use in any other way. It is the application's responsibility to construct the bitmap header and write the file. The following code from the Windowless sample shows how it is done:

BOOL CaptureImage(LPCSTR szFile)
{
    if(pWC) // Our IVMRIWindowlessControl pointer.
    {
        BYTE* lpCurrImage = NULL;

        // Read the current video frame into a byte buffer. The information
        // will be returned in a packed Windows DIB and will be allocated
        // by the VMR.
        if(pWC->GetCurrentImage(&lpCurrImage) == S_OK)
        {
            BITMAPFILEHEADER    hdr;
            DWORD               dwSize, dwWritten;
            LPBITMAPINFOHEADER  pdib = (LPBITMAPINFOHEADER) lpCurrImage;

            // Create a new file to store the bitmap data.
            HANDLE hFile = CreateFile(szFile, GENERIC_WRITE, FILE_SHARE_READ, NULL,
                CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);

            if (hFile == INVALID_HANDLE_VALUE)
                return FALSE;

            // Initialize the bitmap header.
            dwSize = DibSize(pdib);
            hdr.bfType          = BFT_BITMAP;
            hdr.bfSize          = dwSize + sizeof(BITMAPFILEHEADER);
            hdr.bfReserved1     = 0;
            hdr.bfReserved2     = 0;
            hdr.bfOffBits       = (DWORD)sizeof(BITMAPFILEHEADER) + pdib->biSize +
                DibPaletteSize(pdib);

            // Write the bitmap header and bitmap bits to the file.
            WriteFile(hFile, (LPCVOID) &hdr, sizeof(BITMAPFILEHEADER), &dwWritten, 0);
            WriteFile(hFile, (LPCVOID) pdib, dwSize, &dwWritten, 0);

            // Close the file.
            CloseHandle(hFile);

            // The app must free the image data returned from GetCurrentImage().
            CoTaskMemFree(lpCurrImage);

            // Give user feedback that the write has completed.
            MessageBeep(0);
            TCHAR szDir[MAX_PATH];

            GetCurrentDirectory(MAX_PATH, szDir);

            // Strip off the trailing slash, if it exists.
            int nLength = _tcslen(szDir);
            if (szDir[nLength-1] == TEXT('\\'))
                szDir[nLength-1] = TEXT('\0');

            Msg(TEXT("Captured current image to %s\\%s."), szDir, szFile);
            return TRUE;
        }
    }

    return FALSE;
}

Blender: Mixing Multiple Input Streams

Blender is an MFC application that shows the VMR's basic mixing capabilities using its default components. The setup and configuration of the VMR is very similar to the Windowless sample, except that in InitializeWindowlessVMR, after calling SetRenderingMode, we call SetNumberOfStreams with an argument of "2". (This can be any value up to 16.) This method call causes the VMR to sprout two input pins and prepares it to mix and blend that number of video streams in its Mixer-Compositor component, which will pass the composited image to the Allocator-Presenter for final rendering on the screen.

After the VMR is configured, we then call "RenderFile" on two files. We know that if a filter is already present in the graph, the Filter Graph Manager will try to use it first when building a graph automatically. In this case, when the second RenderFile call is made, the Filter Graph Manager sees that the VMR is already present in the graph, and that it has an unconnected input pin, so it uses this same instance of the VMR as the renderer for the second stream. We end up with one graph with two file source filters that, in the case of two Digital Video (DV) AVI files, looks like this.

Blender filter graph image

As an interesting side note, you will notice that each Audio Renderer is shown as a clock source in the graph. When two clocks are equal, as is the case when the same hardware device is underlying two filters, then the Filter Graph Manager will select the most recently added filter as the clock. If the two clocks are not equal, then the Filter Graph Manager will apply some further logic to determine the best one to use. In other words, although you see two clocks in GraphEdit, only one is actually being used by the Filter Graph Manager and that is why the samples "just work" without any special code to specify a clock.

Update the Streams' Size, Position and Alpha Values

The graph is now ready to run, and when the user clicks on the Play button, both video streams begin to play. They are blended together with their rectangle positions and alpha values set to the defaults. The blending is performed by the VMR's Mixer-Compositor, which exposes the IVMRMixerControl interface. The Blender sample provides a user interface that enables the user, at any time during playback, to dynamically change the size and position of the video rectangle of either stream. The individual stream rectangles may be positioned not just anywhere within the visible rectangle, but also completely "offscreen" on any side of the visible rectangle. This combination of the visible area and the surrounding area is called "composition space". The SDK documentation describes the concept of composition space in more detail; the Blender sample shows how to use it in actual practice. In Blender, the positioning is performed manually with a slider control, so the ability to place a video rectangle where it can't be seen may seem at first to be of limited value. But a more sophisticated application could create custom transitions by repositioning the rectangles programmatically in composition space and then sliding them in or out of view in any direction. Note how the function handles mirrored or flipped images using composition space. Here is the code that is called in response to the movement of the slider control in the dialog box:

HRESULT CBlenderDlg::UpdatePinPos(int nStreamID)
{
    HRESULT hr=S_OK;

    // Get a pointer to the selected stream's information.
    STRM_PARAM* p = &strParam[nStreamID];

    // Set the left, right, top, and bottom coordinates.
    NORMALIZEDRECT r = {p->xPos, p->yPos, p->xPos + p->xSize, p->yPos + p->ySize};

    // If mirrored, swap the left/right coordinates in the destination rectangle.
    if (strParam[nStreamID].bMirrored)
    {
        float fLeft = strParam[nStreamID].xPos;
        float fRight = strParam[nStreamID].xPos + strParam[nStreamID].xSize;
        r.left = fRight;
        r.right = fLeft;
    }

    // If flipped, swap the top/bottom coordinates in the destination rectangle.
    if (strParam[nStreamID].bFlipped)
    {
        float fTop = strParam[nStreamID].yPos;
        float fBottom = strParam[nStreamID].yPos + strParam[nStreamID].ySize;
        r.top = fBottom;
        r.bottom = fTop;
    }

    DisplayCoordinates(nStreamID, r);
    // Update the destination rectangle for the selected stream.
    if(pMix) //our IVMRMixerControl pointer
        hr = pMix->SetOutputRect(nStreamID, &r);

    return hr;
}

STRM_PARAM is an application-defined structure that contains the rectangle dimensions and alpha values that will be applied to a video stream. NORMALIZEDRECT is defined in the DirectShow headers; it is used because coordinates to the left of the visible rectangle's x-axis are expressed as negative values. Note that the rectangle position can be updated without stopping the graph. This code could easily be modified to work with a timer to smoothly slide a video rectangle at a given rate into or out of the clipping area. Alpha values could be modified simultaneously by calling IVMRMixerControl::SetAlpha immediately after calling SetOutputRect. In this sample, however, the alpha values are controlled by a separate slider control with a similar message handler.

HRESULT CBlenderDlg::UpdatePinAlpha(int nStreamID)
{
    HRESULT hr=S_OK;
    STRM_PARAM* p = &strParam[nStreamID];
    if(pMix) // Our IVMRMixerControl pointer.
        hr = pMix->SetAlpha(nStreamID, p->Alpha);
    return hr;
}

Watermark: A Logo of Your Own

In the Watermark sample, we see how to add our own alpha-blended bitmap image over the video rectangle, just like those little logos on network television. But of course we can do more than that, so the sample also shows how to create simple animated logos, and perhaps equally importantly, how to turn them off!

To mix in the bitmap that we provide, the VMR needs the Mixer-Compositor component, even if we are only handling one input stream, as is the case in this sample. In our InitializeWindowlessVMR function, we add one line after we put the VMR into windowless mode,

hr = pConfig->SetNumberOfStreams(1);

where pConfig is our IVMRMixerConfig pointer. This forces the mixer component to be loaded. The only other additional line of code beyond what we saw in the Windowless sample is a QueryInterface call to obtain the IVMRMixerBitmap pointer. This, as you might have guessed, is the interface we will use to set and control the bitmap. In the PlayMovieinWindow function, after obtaining the filter graph interfaces, the application calls BlendApplicationImage, which sets up the bitmap's initial position and alpha values within the video rectangle. At the end of the function, IVMRMixerBitmap::SetAlphaBitmap is called to mix the bitmap with the video.

VMRWatermark contains all its bitmap-related functions in the bitmap.h and bitmap.cpp modules. A look at BlendApplicationImage shows that there is no special magic to working with bitmaps and the VMR. It should all be very familiar if you have ever worked with bitmaps in Windows.

HRESULT BlendApplicationImage(HWND hwndApp)
{
    LONG cx, cy;
    HRESULT hr;
    RECT rc={0};

    // Read the default video size.
    hr = pWC->GetNativeVideoSize(&cx, &cy, NULL, NULL);
    if (FAILED(hr))
      return hr;

    // Load the multi-image bitmap to alpha blend from the resource file.
    HBITMAP hbm = LoadBitmap(ghInst, MAKEINTRESOURCE(IDR_VMR_WIDE));

    BITMAP bm;
    HBITMAP hbmOld;
    HDC hdc = GetDC(hwndApp);
    HDC hdcBmp = CreateCompatibleDC(hdc);
    ReleaseDC(hwndApp, hdc);

    GetObject(hbm, sizeof(bm), &bm);
    hbmOld = (HBITMAP)SelectObject(hdcBmp, hbm);

    // Configure the VMR's bitmap structure.
    VMRALPHABITMAP bmpInfo;
    ZeroMemory(&bmpInfo, sizeof(bmpInfo) );
    bmpInfo.dwFlags = VMRBITMAP_HDC;
    bmpInfo.hdc = hdcBmp;

    // The wide bitmap contains five similar images that can be
    // cycled to provide the illusion of animation.
    g_nBitmapWidth = bm.bmWidth;
    g_nImageWidth  = bm.bmWidth / NUM_IMAGES_IN_BITMAP;

    // Display the bitmap in the bottom right corner.
    // rSrc specifies the source rectangle in the GDI device context.
    // To enable animating between multiple single images within a wide 
    // bitmap, we must specify the entire rectangle of the wide image.
    // The VMR will convert this rectangle into a DirectDraw surface,
    // within which we can select a smaller source rectangle to display.
    SetRect(&rc, 0, 0, g_nBitmapWidth, bm.bmHeight);
    bmpInfo.rSrc = rc;

    // rDest specifies the destination rectangle in composition space (0.0f to 1.0f).
    bmpInfo.rDest.left = (float)(cx - g_nImageWidth) / (float)cx - EDGE_BUFFER;
    bmpInfo.rDest.top = (float)(cy - bm.bmHeight) / (float)cy - EDGE_BUFFER;
    bmpInfo.rDest.right = 1.0f - EDGE_BUFFER;
    bmpInfo.rDest.bottom = 1.0f - EDGE_BUFFER;

    // Copy initial settings to global memory for later modification.
    g_rDest = bmpInfo.rDest;
    g_rSrc  = bmpInfo.rSrc;

    // Save the ratio of the bitmap's width to the width of the video file.
    // This value is used to reposition the bitmap in composition space.
    g_fBitmapCompWidth = (float)g_nImageWidth / (float)cx;

    // Transparency value 1.0 is opaque, 0.0 is transparent.
    // For initially setting the bitmap, we'll make it transparent
    // because we need to give the VMR the entire wide image.
    bmpInfo.fAlpha = 0.0;

    // Set the COLORREF so that the bitmap outline will be transparent.
    SetColorRef(bmpInfo);

    // Give the bitmap to the VMR. Since the alpha value is 0, nothing will
    // be displayed yet on the screen, but the VMR will have the information
    // that it needs to allow us to modify the bitmap.
    hr = pBMP->SetAlphaBitmap(&bmpInfo);
    if (FAILED(hr))
        Msg(TEXT("SetAlphaBitmap FAILED!  hr=0x%x\r\n"), hr);

    // Clean up GDI resources.
    DeleteObject(SelectObject(hdcBmp, hbmOld));
    DeleteObject(hbm);
    DeleteDC(hdcBmp);

    // Now change the size of the source rectangle to a single
    // image width. Update the alpha so that the image will be
    // properly displayed.
    SetRect(&rc, 0, 0, g_nImageWidth, bm.bmHeight);
    bmpInfo.rSrc = rc;

    // Save the single-image rectangle for later reference during animation.
    g_rSrcSingle = rc;

    // Set the necessary flags to update the source rectangle.
    bmpInfo.dwFlags = VMRBITMAP_SRCRECT | VMRBITMAP_SRCCOLORKEY;

    // Transparency value 1.0 is opaque, 0.0 is transparent.
bmpInfo.fAlpha = TRANSPARENCY_VALUE;

    // Update the source rectangle and alpha values of the bitmap.
    // Now the image will appear properly on the screen.
    hr = pBMP->UpdateAlphaBitmapParameters(&bmpInfo);
    if (FAILED(hr))
        Msg(TEXT("UpdateAlphaBitmapParameters FAILED!  hr=0x%x\r\n"), hr);

    return hr;
}

For the various special effects, we set up a Windows timer and in our callback methods we basically call IVMRAlphaBitmap::GetAlphaBitmapParameters to retrieve the current alpha and rectangle values, modify those values, then call IVMRAlphaBitmap::UpdateAlphaBitmapParameters to immediately update the bitmap. There is no need to call SetAlphaBitmap as long as the original bitmap isn't being replaced. In fact, SetAlphaBitmap is a relatively expensive call and should be used as sparingly as possible. The MirrorWatermark, FlipWatermark and AnimateWatermark functions all demonstrate this basic concept. AnimateWatermark is the most interesting. It is designed to work with a bitmap that is a series of images like a strip of film. Each time the timer event is fired, the source rectangle for the bitmap is set to contain the next image to the right in the image strip. By using this technique we can avoid having to load a new resource each time this method is called. After calling GetAlphaBitmapParameters, AnimateWatermark calls HandleAnimation to reset the source rectangle for the bitmap. The HandleAnimation code looks like this:

void HandleAnimation(void)
{
    HRESULT hr;
    VMRALPHABITMAP bmpInfo={0};
    static int nCycle=0;

    // Fill the rDest and fAlpha values in bmpInfo.
    hr = pBMP->GetAlphaBitmapParameters(&bmpInfo);

    // Move the image source to the right by one image width.
    bmpInfo.rSrc.left   = g_nImageWidth * nCycle;
    bmpInfo.rSrc.right  = bmpInfo.rSrc.left + g_nImageWidth;
    bmpInfo.rSrc.top    = g_rSrc.top;
    bmpInfo.rSrc.bottom = g_rSrc.bottom;
    nCycle++;

    // If we have passed the last image in the wide bitmap,
    // then reset to the default source location (leftmost image).
    if (bmpInfo.rSrc.left >= g_nBitmapWidth)
    {
        bmpInfo.rSrc = g_rSrcSingle;
        nCycle = 1;
    }

    // Set the necessary flag to update the source rectangle.
    bmpInfo.dwFlags = VMRBITMAP_SRCRECT;

    // Set the COLORREF so that the bitmap outline will be transparent.
    SetColorRef(bmpInfo);

    // Give the VMR a new bitmap to display.
    hr = pBMP->UpdateAlphaBitmapParameters(&bmpInfo);
    if (FAILED(hr))
        Msg(TEXT("UpdateAlphaBitmapParameters FAILED!  hr=0x%x\r\n"), hr);

}

Text: Using the Bitmap for Subtitles

In the Text sample we use many of the same basic bitmap-manipulation routines as in Watermark, but instead of displaying an image, we display text. This technique can be used to apply titles and subtitles to video streams at run time. For the sake of simplicity, the strings and the duration of their display are hard-coded into the application, although the user can select the font. In a later sample, Ticker, we'll see how to dynamically add text through user input. But first, let's look at the basics of using text with the VMR's bitmap image support. All the new code is located in the project's bitmap.cpp module.

The text is updated periodically through the UpdateText function which is called in response to a timer event. UpdateText contains an array of text strings, and increments the pointer to a new string each time the function is called. Inside this function, the BlendText function is called, and this is where the real work is done. Basically, what we need to do is determine the size that our bitmap will be based on the video size and the length of the text string, then create the bitmap, set the text color, and use DrawText to blit the string into the bitmap. Then we position the bitmap in the rectangle by mapping the source rectangle in the device context to the destination rectangle (expressed in composition space coordinates) used by the VMR. Finally, we just call SetAlphaBitmap to apply the new bitmap immediately to the video rectangle. Note that we do not call UpdateAlphaBitmapParameters because we have an entirely new bitmap for each string.

The code listing is rather long, but it is included here because it provides a good example of how GDI functions can be used to supply text overlays to the VMR.

HRESULT BlendText(HWND hwndApp, TCHAR *szNewText)
{
LONG cx, cy;
    HRESULT hr;

    // Read the default video size
    hr = pWC->GetNativeVideoSize(&cx, &cy, NULL, NULL);
    if (FAILED(hr))
    {
        Msg(TEXT("GetNativeVideoSize FAILED!  hr=0x%x\r\n"), hr);
        return hr;
    }

    // Create a device context compatible with the current window
    HDC hdc = GetDC(hwndApp);
    HDC hdcBmp = CreateCompatibleDC(hdc);

    // Write with a known font by selecting it into our HDC
    HFONT hOldFont = (HFONT) SelectObject(hdcBmp, g_hFont);

    // Determine the length of the string, then determine the
    // dimensions (in pixels) of the character string using the
    // currently selected font.  These dimensions are used to create
    // a bitmap below.
    int nLength, nTextBmpWidth, nTextBmpHeight;
    SIZE sz={0};
    nLength = _tcslen(szNewText);
    GetTextExtentPoint32(hdcBmp, szNewText, nLength, &sz);
    nTextBmpHeight = sz.cy;
    nTextBmpWidth  = sz.cx;

    // Create a new bitmap that is compatible with the current window
    HBITMAP hbm = CreateCompatibleBitmap(hdc, nTextBmpWidth, nTextBmpHeight);
    ReleaseDC(hwndApp, hdc);

    // Select our bitmap into the device context and save the old one
    BITMAP bm;
    HBITMAP hbmOld;
    GetObject(hbm, sizeof(bm), &bm);
    hbmOld = (HBITMAP)SelectObject(hdcBmp, hbm);

    // Set initial bitmap settings
    RECT rcText;
    SetRect(&rcText, 0, 0, nTextBmpWidth, nTextBmpHeight);
    SetBkColor(hdcBmp, RGB(255, 255, 255)); // Pure white background
    SetTextColor(hdcBmp, g_rgbColors);   

    // Draw the requested text string onto the bitmap
    TextOut(hdcBmp, 0, 0, szNewText, nLength);

    // Configure the VMR's bitmap structure
    VMRALPHABITMAP bmpInfo;
    ZeroMemory(&bmpInfo, sizeof(bmpInfo) );
    bmpInfo.dwFlags = VMRBITMAP_HDC;
    bmpInfo.hdc = hdcBmp;  // DC which has selected our bitmap

    // Remember the width of this new bitmap
    g_nImageWidth = bm.bmWidth;

    // Save the ratio of the bitmap's width to the width of the video file.
    // This value is used to reposition the bitmap in composition space.
    g_fBitmapCompWidth = (float)g_nImageWidth / (float)cx;

    // Display the bitmap in the bottom right corner.
    // rSrc specifies the source rectangle in the GDI device context 
    // rDest specifies the destination rectangle in composition space (0.0f to 1.0f)
    bmpInfo.rDest.left = 0.0f + X_EDGE_BUFFER;
    bmpInfo.rDest.right = 1.0f - X_EDGE_BUFFER;
    bmpInfo.rDest.top = (float)(cy - bm.bmHeight) / (float)cy - Y_EDGE_BUFFER;
    bmpInfo.rDest.bottom = 1.0f - Y_EDGE_BUFFER;
    bmpInfo.rSrc = rcText;

    // Transparency value 1.0 is opaque, 0.0 is transparent.
    bmpInfo.fAlpha = TRANSPARENCY_VALUE;

    // Set the COLORREF so that the bitmap outline will be transparent
    SetColorRef(bmpInfo);

    // Give the bitmap to the VMR for display
    hr = pBMP->SetAlphaBitmap(&bmpInfo);
    if (FAILED(hr))
        Msg(TEXT("SetAlphaBitmap FAILED!  hr=0x%x\r\n\r\n")
            TEXT("NOTE: Your display must be configured for 32-bit color depth\r\n")
            TEXT("for this sample to display alpha-blended text."), hr);

    // Select the initial objects back into our device context
    DeleteObject(SelectObject(hdcBmp, hbmOld));
    SelectObject(hdc, hOldFont);

    // Clean up resources
    DeleteObject(hbm);
    DeleteDC(hdcBmp);

    return hr;}

Ticker: Scrolling Dynamic Text on Video

Ticker extends the Text sample by providing horizontally scrolling text across the bottom of the screen. There are two modes available through the menu. Static text just displays a prewritten bitmap, which is loaded from the resource file. Dynamic text mode displays user-specified text, including the ability to set font and color. The scrolling effect is obtained by repositioning the bitmap in response to a timer event set by the application in the SlideWatermark, HandleSlide and ResetBitmapPosition functions. Here is the listing for the HandleSlide function, which is where most of the work is done:

void HandleSlide(void)
{
    HRESULT hr;
    VMRALPHABITMAP bmpInfo={0};

    hr = pBMP->GetAlphaBitmapParameters(&bmpInfo);
    if (FAILED(hr))
         Msg(TEXT("GetAlphaBitmapParameters FAILED!  hr=0x%x\r\n"), hr);

    // Slowly decrease the X coordinate
    bmpInfo.rDest.left  -= SLIDE_VALUE;   
    bmpInfo.rDest.right -= SLIDE_VALUE;

    // Once the bitmap disappears off the left side of the screen,
    // reset to the rightmost side of the window.
    if ((bmpInfo.rDest.right <= EDGE_BUFFER))
    {
        bmpInfo.rDest.left = 1.0f;
        bmpInfo.rDest.right = 1.0f + g_fBitmapCompWidth;
    }

    // Set the COLORREF so that the bitmap outline will be transparent
    SetColorRef(bmpInfo);

    hr = pBMP->UpdateAlphaBitmapParameters(&bmpInfo);
    if (FAILED(hr))
            Msg(TEXT("UpdateAlphaBitmapParameters FAILED!  hr=0x%x\r\n"), hr);
}

Advanced Series

The next two samples are considered "advanced" because they involve the use of custom Allocator-Presenters or Mixer-Compositors. VMRMulti shows the kinds of effects you can achieve when you provide your own Allocator-Presenter. Cube also demonstrates cool D3D effects, but it does so by providing a custom Mixer-Compositor to mix three video streams onto a spinning cube.

Cube: Mixing Streams Onto a D3D Surface

The Cube sample shows a relatively simple way to integrate D3D and video, but it has some limitations. First, the streams cannot be stopped or started independently because they are all controlled by the same filter graph manager. Second, the composition space is limited to the size of the largest rectangle in the input streams after it has been aspect-ratio corrected. That means you cannot stretch the video beyond its native size. The VMRMulti sample has neither limitation, but of course it is a more complex application.

Cube – Mixing Streams Onto a D3D Surface

This sample shows how to implement an application-provided mixer-compositor that uses Direct3D 7 APIs to show multiple video streams on the faces of a spinning cube.

Note We use Version 7 because DirectX 8, although it has a vastly superior 3D graphics pipeline compared to DirectX 7, does not provide all the 2D video capabilities required by the VMR. The VMR therefore uses DirectDraw 7 internally, which means that any application-provided compositor or custom Allocator-Presenter must also use version 7.

The approach is quite straightforward. When the application runs, it prompts the user to select up to three files. After the files are selected, the application creates a filter graph, adds the VMR and configures it for the correct number of streams, and then instructs it to use our custom compositor object by calling the IVMRFilterConfig::SetImageCompositor method. The application calls RenderFile for each file. We end up with one filter graph with up to three sources all connected through their intermediary filters to the same VMR, similar to the graph in the Blender sample, and the graph is ready to run.

Our custom compositor implements the IVMRImageCompositor interface, so the VMR will simply call the interface methods at various points while the graph is running to composite the frame before sending the final result to the default Allocator-Presenter. When the VMR is first connected to an upstream filter, it calls SetStreamMediaType to inform the compositor about the details of the video it will passing to it. When the first VMR calls our InitCompositionTarget method, it passes in pointers to the Direct3D object and the surface onto which we will draw the composited video frame. We use these pointers to initialize the rendering environment that we will use for the lifetime of the filter graph. (Note that it is the VMR, not our application, that creates and manages the Direct3D objects.) When the subsequent VMRs call SetStreamMediaType, if the video rectangle is larger than any previous rectangle, then that VMR will call TermCompositionTarget to remove the old composition target surface, and then call InitCompositionSurface to create a new surface that can accommodate the new rectangle.

The actual work of compositing the multiple streams onto the cube is performed inside the CompositeImage function, which is called by the VMR once for each video frame. Inside this function, we calculate the amount of rotation to apply to the cube, calculate the cube's size based on the video rectangles, create the cube's frame, and then apply its surfaces, which can be either one of our video input frames, or else the default background texture (since not all sides of the cube contain video). When this method returns, the VMR then passes the surface to its Allocator-Presenter for final rendering to video memory. When the last frame is rendered, the VMR calls TermCompositionTarget to allow our compositor object to perform any necessary cleanup.

VMRMulti: The Ultimate VMR Sample

The VMRMulti sample, with its independent control of multiple video streams, and cool alpha-blended 3D special effects in the background, provides the most dramatic demonstration of what's possible when you take control of the video rendering process. As you might expect, this is also the most complex code sample, making use of DirectDraw as well as Microsoft Direct3D®.

Note: This sample applies only to the VMR-7.

To run the application, open two or more video files using the Add button. A separate playback window will appear next to the control window. You will see the video rectangles positioned and resized on the screen in a consistent pattern. The D3D effects in the background are randomly generated from a group of texture files. There is always one stream designated as the primary stream; this is the video in the large rectangle. Double click on another rectangle or on a file in the file list to make it the primary stream. All video streams loop continuously. 

The first thing to mention about this application is the unusual filter graph configuration it creates. Each video stream is its own separate filter graph complete with its own filter graph manager and instance of a VMR, and all of these VMRs share a common Allocator-Presenter provided by the application. It is this Allocator-Presenter that controls the sizing and position of the video rectangles and composites each video streams over the background D3D effects. The following diagram shows the relationship of the VMRs in the filter graphs to the application-provided Allocator-Presenter, as well as to the default Alocator-Presenter instantiated separately by the application's object.

VMRMulti filter graph image

Why do we go to all this trouble instead of simply creating a single graph with a single VMR with multiple input pins? The short answer is control. If we were to connect our multiple streams up to a single VMR with multiple input pins, we would not be able to obtain a separate IMediaSeeking interface for each video stream. Also, we would not be able to remove files from the mix without stopping the graph because hardly any source filters support the IPinConfig interface required for dynamic graph building. The ability to add and remove streams creates many more possibilities for interesting video compositions.

Set up the Allocator-Presenter and the DirectDraw and D3D Environments

The application's Allocator-Presenter component is implemented in the CMultiSAP class, which is instantiated as a global variable in the VideoMultiVMR_OnCreate function that is called when the application's control window is first created. Note that this occurs before any filter graph is constructed. After the object is instantiated, the CMultiSAP::Initialize function is called, which in turn calls CMultiSAP::InitializeEnvironment where the real setup action takes place.

First the CreateDefaultAllocatorPresenter function is called, which uses the DirectShow-defined CLSID_AllocPresenter to co-create a private instance of the VMR's default allocator-presenter and receive its IVMRSurfaceAllocator pointer. This instance of the VMR will not be used in any filter graph; it is only used so that our CMultiSAP object can call IVMRSurfaceAllocator::AllocateSurface to obtain a pointer to the returned DirectDraw surface. This surface won't be used for rendering; it is used only to obtain a pointer to the DirectDraw object that is associated with that surface. We need this DirectDraw object to set up the rest of our D3D and DirectDraw environment in CMultiSAP::InitializeEnvironment. Also note that in CreateDefaultAllocatorPresenter we use the IVMRSurfaceAllocator interface to QueryInterface for the IVMRWindowlessControl interface and then call SetVideoClippingWindow. This is done before the call to AllocateSurface so that the DirectDraw object we ultimately obtain is associated with our window. We also use our default Allocator-Presenter to perform some low-level housekeeping related to video memory by forwarding all IVMRImagePresenter interface calls to it, even though we also use our own implementation of this interface. This process is described in more detail later.

The code listing for InitializeEnvironment is rather long, so we won't show it here. Just note that all the 3D rendering happens in the CD3DHelper and CSparkle classes, so these are the classes you can modify to create your own effects. It is beyond the scope of this article to discuss the details of how the D3D special effects are created; this is left as an exercise for the reader.

Build the Filter Graphs

Once the Allocator-Presenter object and the D3D environment have been initialized, the application is ready to build its filter graphs. CMultiSAP contains a list of CMovie objects, each of which represents a separate filter graph. In CMovie you will find all the familiar VMR graph-building techniques discussed in the previous samples; most of the work is done in the OpenMovie and AddVideoMixingRenderertoFG member functions. One difference is that we have already associated the DirectDraw object with our window, so there is no need to QueryInterface on the VMRs in each graph for their IVMRWindowlessControl interfaces. We need to tell each VMR in the graph to use our custom Allocator-Presenter, which is the global instance of the CMultiSAP class. This is done in AddVideoMixingRenderertoFG by calling QueryInterface using the VMR's IBaseFilter interface to obtain the VMR's IVMRSurfaceAllocatorNotify interface. We then call the IVMRSurfaceAllocatorNotify::AdviseSurfaceAllocator method and pass in a pointer to the CMultiSAP object. Then we call IVMRSurfaceAllocatorNotify::SetDDrawDevice to specify the DirectDraw device that will be used for rendering.

Our CMultiSAP object implements the IVMRSurfaceAllocator interface, whose methods will be called by each VMR in the list. The VMR calls the PrepareSurface, AllocateSurface and FreeSurface methods to set up and free the video memory when the graph is created and destroyed. (The AdviseNotify method is not used because CMultiSAP is using a separate default Allocator-Presenter for its default handling, as discussed later.)

Composite and Render the Frames

CMultiSAP implements IVMRImagePresenter, whose PresentImage method is called by each VMR in the filter graph whenever it has received a complete sample from the upstream decoder. When the CMultiSAP object receives a PresentImage call from one of the VMRs, it copies the data in the supplied DirectDraw surface to the texture surface in the corresponding CMovie object. We must make a data copy here; we cannot simply use the pointer to the VMR's sample itself because that could result in deadlocks in the filter graph.

So, at this point the CMultiSAP object contains a list of CMovie objects, some or all of which have DirectDraw texture surfaces containing video data. These surfaces still need to be composited and rendered onto one of our back buffers in video memory. CMultiSAP performs its rendering by creating a worker thread that functions as a timer and calls the CMultiSAP::ComposeandRender function at regular intervals. In this application the interval is set at 30 milliseconds, or approximately 30 frames per second, although you could modify this value, or set it to match the frame rate of the input streams. In the code listing for the following timer function, note that before we actually tell CMultiSAP to begin rendering, we pass through a call to IVMRImagePresenter::StartPresenting on the default Allocator-Presenter. Likewise, after CMultiSAP is done with its rendering work and ComposeAndRender returns, we again call the default Allocator-Presenter, this time with StopPresenting. This technique of using a default Allocator-Presenter is described more fully in the DirectShow SDK documentation. The worker thread function looks like this:

DWORD CMultiSAP::ComposeThread()
{
    timeBeginPeriod(1);
    const DWORD dwFrameRate = 30;
    
    DWORD dwFramePeriod = 1000 / dwFrameRate;
    DWORD dwStartTime = timeGetTime();
    
    m_pPresenter->StartPresenting(0);
    
    for ( ;; ) {
        
DWORD dwTimeThisFrame = dwStartTime + 
                       ((m_dwFrameNum++ * 1000) / dwFrameRate);
    long lWaitTime = dwTimeThisFrame - timeGetTime();
   if (lWaitTime < 0) {
        // lWaitTime = 0;
        // OutputDebugStringA("Skipping a frame\n");
        continue;
        }
        
    DWORD rc = WaitForSingleObject(m_hQuitEvent, (DWORD)lWaitTime);
    if (rc == WAIT_OBJECT_0) {
        break;
    }
        
#ifdef SPARKLE
        if (m_pSparkle)
            m_pSparkle->RenderFrame();
#endif
        
    ComposeAndRender();
    }
    
    m_pPresenter->StopPresenting(0);
    timeEndPeriod(1);
    return 0;
}

ComposeAndRender performs three basic tasks. First it selects the D3D effect and uses the various D3D objects (CD3dHelper, CEffect, and others) to set up and render the D3D effect for the entire video rectangle. It then determines the destination rectangle for each movie in its list, and uses the CD3DHelper::RenderFrame method to composite each video rectangle over the D3D effects. After all the movies in the list have been rendered in this way, CMultiSAP once again passes a call through to its default Allocator-Presenter, this time calling its IVMRImagePresenter::PresentImage method, which does some low-level housekeeping work related to video memory and the frame buffers. But even though we use the default Allocator-Presenter for these tasks, it is important to understand that it is our own CMultiSAP object that is doing the actual rendering of data onto video memory.

When ComposeAndRender is done, the calling function has finished its cleanup work as shown in the previous code listing, it starts its loop again and waits for another 30 milliseconds before starting the entire process over again. In the meantime, the CMovie objects have been busy obtaining new frames from the VMRs, but this activity is not synchronized with the ComposeAndRender function except in the sense that it is thread-safe. In other words, ComposeAndRender does not check to make sure that each CMovie object actually has new data, nor does it block the CMovie objects from receiving frames if the stream's frame rate is faster than 30 frames per second. CMultiSAP does not gate the delivery of samples from the VMRs in any way. Of course, each VMR is gating its respective filter graph to control the frame rate based on the information in the VIDEOINFOHEADER in the media samples. This independence of the ComposeAndRender function from the frame rates of the filter graphs means that theoretically you could use a VMRMulti type of application to blend multiple sources with different or even variable frame rates. This is also the approach to use when rendering different types of media sources (DVD, TV, still images, etc.).

Summary

In this article we have shown how to configure the VMR, how to use it to capture still images, how to overlay and even scroll text over video, how to mix and blend multiple video streams, and how to integrate video with your own 3D effects. Hopefully, these samples have helped you get started using the new Video Mixing Renderer and given you ideas for new possibilities in your video applications.