Processing Media Data with the Source Reader (Windows)

MSDN Library
Win32 and COM Development
Graphics and Multimedia
Audio and Video
Microsoft Media Foundation SDK
Media Foundation Programming Guide
Source Reader

Processing Media Data with the Source Reader

Tutorial: Decoding Audio

Why Use the Source Reader?

The source reader is an alternative to using the Media Session and the Microsoft Media Foundation pipeline to process media data.

Media Foundation provides a pipeline that is optimized for playback. The pipeline is end-to-end, meaning it handles data flow from the source (such as a video file) all the way to the destination (such as the graphics display). However, if you want to read or modify the data as it goes through the pipeline, you must write a custom plug-in. That requires a fairly deep knowledge of the Media Foundation pipeline. For certain tasks, creating a new plug-in is too much overhead. The source reader is designed for this type of situation, when you want to get the raw data from a source without the overhead of the entire pipeline.

Internally, the source reader holds a pointer to a media source. A media source is a Media Foundation object that generates media data from an external source, such as a media file or video capture device. The source reader manages all of the method calls to the media source. (For more information about media sources, see Media Sources.)

If the media source delivers compressed data, you can use the source reader to decode the data. In that case, the source reader will load the correct decoder and manage the data flow between the media source and the decoder. The source reader can also perform some limited video processing: color conversion from YUV to RGB-32, and software deinterlacing, although these operations are not recommended for real-time video rendering. The following image illustrates this process.

The source reader does not send the data to a destination; it is up to the application to consume the data. For example, the source reader can read a video file, but it will not render the video to the screen. Also, the source reader does not manage a presentation clock, handle timing issues, or synchronize video with audio.

Consider using the source reader when:

You want to get data from a media file without worrying about the underlying file structure.
You want to get data from an audio or video capture device.
Your data-processing tasks are not time sensitive, or you do not require a presentation clock.
You already have a media pipeline that is not based on Media Foundation, and you want to incorporate the Media Foundation media sources into your own pipeline.

The source reader is not recommended in the following situations:

For media playback applications. Instead, use MFPlay (IMFPMediaPlayer) or the Media Session (IMFMediaSession).
For protected content. The source reader does not support digital rights management (DRM).
If you care about the details of the underlying file structure. The source reader hides that type of detail.

To use the source reader, follow these basic steps:

Create an instance of the source reader.
Enumerate the possible output formats.
Set the actual output format for each stream.
Process data from the source.

The remainder of this topic describes these steps in detail.

Creating the Source Reader

To create an instance of the source reader, call one of the following functions:

MFCreateSourceReaderFromURL. This function takes a URL as input. Internally, it uses the Source Resolver to create a media source from the URL.
MFCreateSourceReaderFromByteStream. This function takes a pointer to a byte stream. It uses the source resolver to create a media source.
MFCreateSourceReaderFromMediaSource. This function takes a pointer to a media source, and is useful for media sources that cannot be created by the source resolver, including capture devices or custom media sources.

Each of these functions takes an optional IMFAttributes pointer, which is used to set various options on the source reader, as described in the reference topics for these functions. To get the default behavior, set this parameter to NULL. Each function returns an IMFSourceReader pointer as an output parameter. You must call the MFStartup function before calling any of these functions.

The following code creates the source reader from a URL.

void wmain(int argc, WCHAR* argv[])
{
    const WCHAR *sURL = argv[1];

    HRESULT hr = S_OK;
    IMFSourceReader *pReader = NULL;

    hr = MFStartup(MF_VERSION);

    if (SUCCEEDED(hr))
    {
        hr = MFCreateSourceReaderFromURL(
            sURL,
            NULL,
            &pReader
            );
    }

    if (SUCCEEDED(hr))
    {
        // TODO: Use the source reader. (Not shown.)
    }

    SAFE_RELEASE(pReader);
    MFShutdown();
};

Enumerating Output Formats

Every media source has at least one stream. For example, a video file might contain a video stream and an audio stream. The format of each stream is described using a media type, represented by the IMFMediaType interface. For more information about media types, see Media Types. You must examine the media type to understand the format of the data that you get from the source reader.

Initially, every stream has a default format, which you can find by calling the IMFSourceReader::GetCurrentMediaType method.

For a given stream, the media source offers a list of possible media types for that stream. The number of types depends on the source. If the source represents a media file, there is typically only one media type per stream. A webcam, on the other hand, might be able to stream video in several different formats. In that case, the application can select which format to use from the list of media types.

To get one of the possible media types for a stream, call the IMFSourceReader::GetNativeMediaType method. This method takes two index parameters. The first is the index of the stream, and the second is an index into the list of media types for that stream.

To enumerate all of the types for a given stream, increment the list index while keeping the stream index constant. When the list index goes out of bounds, the GetNativeMediaType method returns MF_E_NO_MORE_TYPES. To enumerate every stream, increment the stream index. When the stream index goes out of bounds, the GetNativeMediaType method returns MF_E_INVALIDSTREAMNUMBER.

The following code shows how to enumerate every media type for every stream.

for (DWORD i = 0; ; i++)
{
    for (DWORD j = 0; ; j++)
    {
        IMFMediaType *pType = NULL;
        
        // Get the j'th media type for the i'th stream.
        hr = pReader->GetNativeMediaType(i, j, &pType);
        if (FAILED(hr))
        {
            break;
        }
        // TODO: Examine the media type.
        
        pType->Release();
    }
    if (hr == MF_E_NO_MORE_TYPES)
    {
        hr = S_OK; // No more types for this stream. Continue.
    }
    else if (hr == MF_E_INVALIDSTREAM)
    {
        hr = S_OK;
        break;   // No more streams. Quit the loop.
    }
    else if (FAILED(hr))
    {
         break;  // Some other error.
    }
}

Setting Output Formats

To change the media type on a stream, call the IMFSourceReader::SetCurrentMediaType method. This method takes a stream index and a media type as input.

To get data directly from the source without decoding it, use one of the types returned by GetNativeMediaType.

To decode the stream, create a new media type that describes the desired uncompressed format. It is usually sufficient to specify the major type (audio or video) and the subtype, as follows:

Call MFCreateMediaType to create a new media type.
Set the MF_MT_MAJOR_TYPE attribute on the media type.
Set the MF_MT_SUBTYPE attribute on the media type.
Call IMFSourceReader::SetCurrentMediaType.

The source reader will automatically load the necessary decoders. To get the complete format details, call IMFSourceReader::GetCurrentMediaType.

Processing Media Data

To get media data from the source, call the IMFSourceReader::ReadSample method, as shown in the following code.

DWORD dwStream = 0;
DWORD dwFlags = 0;
IMFSample *pSample = NULL;

hr = pReader->ReadSample(
    MF_SOURCE_READER_ANY_STREAM,
    0,  // Flags
    &dwStream,
    &dwFlags,
    &pSample
    );

The first parameter is the index of the stream for which you want to get data. You can also specify MF_SOURCE_READER_ANY_STREAM to get the next available data from any stream. The second parameter contains optional flags; see MF_SOURCE_READER_CONTROL_FLAG for a list of these. The third parameter receives the index of the stream that actually produces the data. You will need this information if you set the first parameter to MF_SOURCE_READER_ANY_STREAM. The fourth parameter receives status flags, indicating various events that can occur while reading the data, such as format changes in the stream. For a list of status flags, see MF_SOURCE_READER_FLAG.

If the media source is able to produce data for the requested stream, the last parameter of ReadSample receives a pointer to the IMFSample interface of a media sample object. Use the media sample to:

Get a pointer to the media data.
Get the presentation time and sample duration.
Get attributes that describe interlacing, field dominance, and other aspects of the sample.

The contents of the media data depend on the format of the stream. For an uncompressed video stream, each media sample contains a single video frame. For an uncompressed audio stream, each media sample contains a sequence of audio frames.

The ReadSample method can return S_OK and yet not return a media sample in the pSample parameter. For example, when you reach the end of the file, ReadSample sets the MF_SOURCE_READERF_ENDOFSTREAM flag in dwFlags and sets pSample to NULL. In this case, the ReadSample method returns S_OK because no error has occurred, even though the pSample parameter is set to NULL. Therefore, always check the value of pSample before you dereference it.

Using Asynchronous Mode

The source reader operates either in synchronous mode or asynchronous mode. The code example shown in the previous section assumes that the source reader is using synchronous mode, which is the default. In synchronous mode, the IMFSourceReader::ReadSample method blocks while the media source produces the next sample. A media source typically acquires data from some external source (such as a local file or a network connection), so the method can block the calling thread for a noticeable amount of time.

In asynchronous mode, the ReadSample returns immediately and the work is performed on another thread. After the operation is complete, the source reader calls the application through the IMFSourceReaderCallback callback interface. To use asynchronous mode, you must provide a callback pointer when you first create the source reader, as follows:

Create an attribute store by calling the MFCreateAttributes function.
Set the MF_SOURCE_READER_ASYNC_CALLBACK attribute on the attribute store. The attribute value is a pointer to your callback object.
When you create the source reader, pass the attribute store to the creation function in the pAttributes parameter. All of the functions to create the source reader have this parameter.

The following example shows these steps.

class SourceReaderCB; // Class declaration. (Implementation not shown.)

HRESULT hr = S_OK;
IMFSourceReader *pReader = NULL;
IMFAttributes *pAttributes = NULL;

// Create an instance of the callback object.
SourceReaderCB *pCallback = new (std::nothrow) SourceReaderCB();

if (pCallback == NULL)
{
    hr = E_OUTOFMEMORY;
}


if (SUCCEEDED(hr))
{
    hr = MFCreateAttributes(
        &pAttributes,
        1  // Initialize size.
        );
}

if (SUCCEEDED(hr))
{
    hr = pAttributes->SetUnknown(
        MF_SOURCE_READER_ASYNC_CALLBACK,
        pCallback
        );
}

if (SUCCEEDED(hr))
{
    hr = MFCreateSourceReaderFromURL(
        sURL,
        pAttributes,
        &pReader
        );
}

// TODO: Use the source reader. (Not shown.)

SAFE_RELEASE(pAttributes);
SAFE_RELEASE(pReader);

After you create the source reader, you cannot change the synchronous or asynchronous mode.

To get data in asynchronous mode, call the ReadSample method but set the last three parameters to NULL, as shown in the following example.

hr = pReader->ReadSample(
    MF_SOURCE_READER_ANY_STREAM,
    0,  // Flags
    NULL,
    NULL,
    NULL
    );

When the ReadSample method completes asynchronously, the source reader calls your IMFSourceReaderCallback::OnReadSample method. This method has four parameters:

hrStatus: Contains an HRESULT value. This is the same value that ReadSample would return in synchronous mode. If hrStatus contains an error code, you can ignore the remaining parameters.
dwStreamIndex, dwStreamFlags, and pSample: These three parameters are equivalent to the last three parameters in ReadSample. They contain the stream number, status flags, and IMFSample pointer, respectively.

Getting the File Duration

To get the duration of a media file, call the IMFSourceReader::GetPresentationAttribute method and request the MF_PD_DURATION attribute, as shown in the following code.

HRESULT GetDuration(
    IMFSourceReader *pReader, 
    LONGLONG *phnsDuration
    )
{
    PROPVARIANT var;
    PropVariantInit(&var);

    HRESULT hr = pReader->GetPresentationAttribute( 
        MF_SOURCE_READER_MEDIASOURCE, 
        MF_PD_DURATION, 
        &var 
        );

    if (SUCCEEDED(hr))
    {
        hr = PropVariantToInt64(var, phnsDuration);
    }
 
    PropVariantClear(&var);
    return hr;
}

The function shown here gets the duration in 100-nanosecond units. Divide by 10,000,000 to get the duration in seconds.

Seeking

A media source that gets data from a local file can usually seek to arbitrary positions in the file. Capture devices such as webcams generally cannot seek, because the data is live. A source that streams data over a network might be able to seek, depending on the network streaming protocol.

To find out whether a media source can seek, call IMFSourceReader::GetPresentationAttribute and request the MF_SOURCE_READER_MEDIASOURCE_CHARACTERISTICS attribute, as shown in the following code:

HRESULT GetSourceFlags(IMFSourceReader *pReader, ULONG *pulFlags)
{
    ULONG flags = 0;

    PROPVARIANT var;
    PropVariantInit(&var);

    hr = pReader->GetPresentationAttribute(
        MF_SOURCE_READER_MEDIASOURCE, 
        MF_SOURCE_READER_MEDIASOURCE_CHARACTERISTICS, 
        &var
        );

    if (SUCCEEDED(hr))
    {
        hr = PropVariantToUInt32(var, &flags);
    }
    if (SUCCEEDED(hr))
    {
        *pulFlags = flags;
    }

    PropVariantClear(&var);
    return hr;
}

This function gets a set of capabilities flags from the source. These flags are defined in the MFMEDIASOURCE_CHARACTERISTICS enumeration. Two flags relate to seeking:

Flag	Description
MFMEDIASOURCE_CAN_SEEK	The source can seek.
MFMEDIASOURCE_HAS_SLOW_SEEK	Seeking might take a long time to complete. For example, the source might need to download the entire file before it can seek. (There are no strict criteria for a source to return this flag.)

To seek, call the IMFSourceReader::SetCurrentPosition method, as shown in the following code.

HRESULT SetPosition(
    IMFSourceReader *pReader, 
    const LONGLONG& hnsPosition // Position in 100-nanosecond units.
    )
{
    PROPVARIANT var;
    HRESULT hr = InitPropVariantFromInt64(hnsPosition, &var);

    if (SUCCEEDED(hr))
    {
        hr = pReader->SetCurrentPosition(GUID_NULL, var);
    }

    PropVariantClear(&var);
    return hr;
}

The first parameter gives the time format that you are using to specify the seek position. All media sources in Media Foundation are required to support 100-nanosecond units, indicated by the value GUID_NULL. The second parameter is a PROPVARIANT that contains the seek position. For 100-nanosecond time units, the data type is LONGLONG.

Be aware that not every media source provides frame-accurate seeking. The accuracy of seeking depends on several factors, such as the key frame interval, whether the media file contains an index, and whether the data has a constant or variable bit rate. Therefore, after you seek to a position in a file, there is no guarantee that the time stamp on the next sample will exactly match the requested position. Generally, the actual position will not be later than the requested position, so you can discard samples until you reach the desired point in the stream.

Hardware Acceleration

The source reader is compatible with Microsoft DirectX Video Acceleration (DXVA) 2.0 for hardware accelerated video decoding. To use DXVA with the source reader, perform the following steps.

Create a Microsoft Direct3D device.
Call the DXVA2CreateDirect3DDeviceManager9 function to create the Direct3D device manager. This function gets a pointer to the IDirect3DDeviceManager9 interface.
Call the IDirect3DDeviceManager9::ResetDevice method with a pointer to the Direct3D device.
Create an attribute store by calling the MFCreateAttributes function.
Create the source reader. Pass the attribute store in the pAttributes parameter of the creation function.

When you provide a Direct3D device, the source reader allocates video samples that are compatible with the DXVA video processor API. You can use DXVA video processing to perform hardware deinterlacing or video mixing. For more information, see DXVA Video Processing. Also, if the decoder supports DXVA 2.0, it will use the Direct3D device to perform hardware-accelerated decoding.