Walkthrough: Reader app (DirectX and XAML)

Article
08/31/2015

Apps that incorporate a reading experience are quickly becoming popular as tablet PCs become more prevalent. Here we show you how to create a rich document reader app with DirectX, C++, and XAML. We cover the structure, technologies and best practices for this type of app, including how to use a Virtual Surface Image Source to interoperate between DirectX drawing code and XAML surface management, use DirectWrite to load fonts, use WIC to load images, and to apply Direct2D image effects to images.

We walk through much of the code for the DirectX Magazine app sample with XAML interop and look at the different components that make up the app.

Magazine sample structure

The files in the sample can be split into three groups, XML, XAML, and DirectX. The files in the XML group are responsible for representing each of the tags in the XML file that holds all the info about an article. The files in the XAML group are responsible for setting up the XAML elements and working with the Virtual Surface Image Source. Finally, the files in the DirectX group are responsible for things like the DirectWrite custom font loading, Direct2D image effects, and WIC image decoding.

Name	Type
BindableProperty	XAML
ContentImageSource	XAML
Design	XML
Document	XML
Element	XML
FontFileStream	DirectX
FontLoader	DirectX
Image	XML
ImageFile	XML
ImageFrame	XML
Layer	XML
List	XML
Page	XML
PageModel	XAML
PageRoll	XML
Rectangle	XML
Resource	XML
Story	XML
Text	XML
TextFrame	XML
TreeIterator	XML

Virtual Surface Image Source

Virtual Surface Image Source (VSIS) is a XAML managed rendering surface that is very useful when you want to write an app that involves panning and zooming. This surface is like any other bitmap source in DirectX, except for the fact that XAML manages the interactions between DirectX and the image source.

When you want to use a VSIS in your app, you must:

Determine the size of the content.
Create the VSIS that is the size of the image.
Set the DXGI device as device on the VSIS.
Register the image source callback so you receive messages when you need to render content.

The code here shows how to perform these steps.

    // Measure the content and store its size.
    Measure(&m_contentSize);

    // Create an image source at the initial pixel size.
    m_imageSource = ref new VirtualSurfaceImageSource(m_contentSize.cx, m_contentSize.cy);

    ComPtr<IUnknown> unknown(reinterpret_cast<IUnknown*>(m_imageSource));
    unknown.As(&m_imageSourceNative);

    auto renderer = m_document->GetRenderer();

    // Set DXGI device to the image source
    ComPtr<IDXGIDevice> dxgiDevice;
    renderer->GetDXGIDevice(&dxgiDevice);
    m_imageSourceNative->SetDevice(dxgiDevice.Get());

    // Register image source's update callback so update can be made to it.
    m_imageSourceNative->RegisterForUpdatesNeeded(this);

Now, you have a VSIS and your class is registered to receive callback info from the virtual surface. Next, you must implement the callback methods that the VSIS calls when you need to update the content.

When the user manipulates the VSIS by scrolling through its content, the VSIS calls the UpdatesNeeded method of your registered class. So, you must implement the UpdatesNeeded callback method.

The code here shows you how to implement the UpdatesNeeded callback method and the Draw helper method. When the VSIS calls this callback method on the ContentImageSource class, the method retrieves the updated rect that the VSIS is rendering using the **VSISGetUpdateRectCount method. You then call into the draw method with this updated area.

// This method is called when the framework needs to update region managed by
// the virtual surface image source.
HRESULT STDMETHODCALLTYPE ContentImageSource::UpdatesNeeded()
{
    HRESULT hr = S_OK;

    try
    {
        ULONG drawingBoundsCount = 0;

        DX::ThrowIfFailed(
            m_imageSourceNative->GetUpdateRectCount(&drawingBoundsCount)
            );

        std::unique_ptr<RECT[]> drawingBounds(new RECT[drawingBoundsCount]);

        DX::ThrowIfFailed(
            m_imageSourceNative->GetUpdateRects(drawingBounds.get(), drawingBoundsCount)
            );

        // This code doesn't try to coalesce multiple drawing bounds into one. Although that
        // extra process  reduces the number of draw calls, it requires the virtual surface
        // image source to manage non-uniform tile size, which requires it to make extra copy
        // operations to the compositor. By using the drawing bounds it directly returns, which are
        //  non-overlapping  tiles of the same size, the compositor can use these tiles directly,
        // which can greatly reduce the amount of memory needed by the virtual surface image source.
        // This results in more draw calls, but Direct2D can accommodate them
        // without significant impact on presentation frame rate.
        for (ULONG i = 0; i < drawingBoundsCount; ++i)
        {
            if (Draw(drawingBounds[i]))
            {
                // Drawing isn't complete. This can happen when the content is still being
                // asynchronously loaded. Inform the image source to invalidate the drawing
                // bounds so that it calls back to redraw.
                DX::ThrowIfFailed(
                    m_imageSourceNative->Invalidate(drawingBounds[i])
                    );
            }
        }
    }
    catch (Platform::Exception^ exception)
    {
        hr = exception->HResult;
    }

    return hr;
}

bool ContentImageSource::Draw(RECT const& drawingBounds)
{
    ComPtr<IDXGISurface> dxgiSurface;
    POINT surfaceOffset = {0};

    DX::ThrowIfFailed(
        m_imageSourceNative->BeginDraw(
            drawingBounds,
            &dxgiSurface,
            &surfaceOffset
            )
        );

    auto renderer = m_document->GetRenderer();

    ComPtr<ID2D1DeviceContext> d2dDeviceContext;
    renderer->GetD2DDeviceContext(&d2dDeviceContext);

    ComPtr<ID2D1Bitmap1> bitmap;
    DX::ThrowIfFailed(
        d2dDeviceContext->CreateBitmapFromDxgiSurface(
            dxgiSurface.Get(),
            nullptr,
            &bitmap
            )
        );

    // Begin the drawing batch
    d2dDeviceContext->BeginDraw();

    // Scale content design coordinate to the display coordinate,
    // then translate the drawing to the designated place on the surface.
    D2D1::Matrix3x2F transform =
        D2D1::Matrix3x2F::Scale(
            m_document->DesignToDisplayWidth(1.0f),
            m_document->DesignToDisplayHeight(1.0f)
            ) *
        D2D1::Matrix3x2F::Translation(
            static_cast<float>(surfaceOffset.x - drawingBounds.left),
            static_cast<float>(surfaceOffset.y - drawingBounds.top)
            );

    // Prepare to draw content. This is the appropriate time for content element
    // to draw to an intermediate if there is any. It is important for performance
    // reason that you don't call SetTarget too often. Preparing the intermediates
    // upfront reduces the number of times the render target switches back and forth.
    bool needRedraw = m_content->PrepareToDraw(
        m_document,
        transform
        );

    if (!needRedraw)
    {
        // Set the render target to surface given by the framework
        d2dDeviceContext->SetTarget(bitmap.Get());

        d2dDeviceContext->SetTransform(D2D1::IdentityMatrix());

        // Constrain the drawing to the designated portion of the surface
        d2dDeviceContext->PushAxisAlignedClip(
            D2D1::RectF(
                static_cast<float>(surfaceOffset.x),
                static_cast<float>(surfaceOffset.y),
                static_cast<float>(surfaceOffset.x + (drawingBounds.right - drawingBounds.left)),
                static_cast<float>(surfaceOffset.y + (drawingBounds.bottom - drawingBounds.top))
                ),
            D2D1_ANTIALIAS_MODE_ALIASED
            );

        // The Clear call must follow the PushAxisAlignedClip call.
        // Placing the Clear call before the clip is set violates the contract of the
        // virtual surface image source in that the app draws outside the
        // designated portion of the surface the image source hands over to it. This
        // violation won't actually cause the content to spill outside the designated
        // area because the image source will safeguard it. But this extra protection
        // has a runtime cost associated with it, and in some drivers this cost can be
        // very expensive. So the best performance strategy here is to never create a
        // situation where this protection is required. Not drawing outside the appropriate
        // clip does that the right way.
        d2dDeviceContext->Clear(D2D1::ColorF(0, 0));

        // Draw the content
        needRedraw = m_content->Draw(
            m_document,
            transform
            );

        d2dDeviceContext->PopAxisAlignedClip();

        d2dDeviceContext->SetTarget(nullptr);
    }

    // End the drawing
    DX::ThrowIfFailed(
        d2dDeviceContext->EndDraw()
        );

    // Submit the completed drawing to the framework
    DX::ThrowIfFailed(
        m_imageSourceNative->EndDraw()
        );

    return needRedraw;
}

With the callbacks implemented, your app now receives info from the VSIS about the region that you need to draw. Your draw function just needs to get this rect, and draw to the VSIS in the right area. By implementing these two callbacks, and registering the class with the VSIS, the app can render to the virtual surface and update the necessary area when the user manipulates the content to reveal more of the surface.

Content loading and data binding with XML

The info that our app displays in the magazine is all stored in the Sample.story file. This file is an XML document that contains info about the article content, header, font face, background image, and other attributes. Each important piece of info that the app needs to digest has a tag. By using the tags, you can easily expand the articles, create objects from that tagged info, and bind that data to the XAML in your app.

Because the info in the .story file is in a custom XML schema, the class in TreeIterator.h helps parse this XML data and puts it into the specialized classes of the app.

Here is an example of the info that’s held in the .story file the app uses.

<?xml version="1.0" encoding="utf-8"?>
<story>
    <resource>
        <image name="grasshopper-image" file="grasshopper.jpg"/>
        <image name="butterfly-image" file="butterfly.jpg"/>
        <text name="section" font-family="Arial" font-stretch="2" font-size="22">A BUG'S LIFE</text>
        <text name="grasshopper-title" font-family="Gill Sans" font-stretch="3" font-weight="700" font-size="70">GRASSHOPPER</text>
        <text name="grasshopper-quote" font-family="Pericles" font-size="18">a slender plant-eating flying and jumping insect that produces a buzzing sound by rubbing its back legs against its forewings</text>
        <text name="grasshopper-body" font-family="Kootenay" font-size="16"><bold>LOREM</bold> ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui.
In hac habitasse platea dictumst. Curabitur at lacus ac velit ornare lobortis. Curabitur a felis in nunc fringilla tristique. Morbi mattis ullamcorper velit. Phasellus gravida semper nisi. Nullam vel sem. Pellentesque libero tortor, tincidunt et, tincidunt eget, semper nec, quam. Sed hendrerit. Morbi ac felis. Nunc egestas, augue at pellentesque laoreet, felis eros vehicula leo, at malesuada velit leo quis pede. Donec interdum, metus et hendrerit aliquet, dolor diam sagittis ligula, eget egestas libero turpis vel mi. Nunc nulla. Fusce risus nisl, viverra et, tempor et, pretium in, sapien. Donec venenatis vulputate lorem.</text>

As the XML snippet shows, the tags for the info correspond directly to some of the XML classes that in the app. When the iterator parses the XML file, for each tag, a class of the same name is created and stores the info from within the tag.

After we have all our info out of the XML file and into the app, you bind the data to elements on the UI. This data binding exists in two places, in the XAML markup for the page, and in the cpp file that accompanies the XAML markup page. In this example, it exists in MainPage.xaml and MainPage.cpp.

In the XAML markup create instances of the FlipView, DataTemplate, and ScrollViewer classes. Then, bind the info to those XAML markup items. This is the XAML markup code for these elements.

<FlipView x:Name="FlipView" ManipulationMode="TranslateInertia">
        <FlipView.ItemTemplate>
            <DataTemplate>
                <ScrollViewer ZoomMode="Disabled" VerticalScrollMode="Enabled" IsVerticalRailEnabled="True">
                    <ScrollViewer.Background>
                        <ImageBrush ImageSource="{Binding Background}"/>
                    </ScrollViewer.Background>
                    <Image 
                        Source="{Binding Content}" 
                        Width="{Binding ContentWidth}" 
                        Height="{Binding ContentHeight}" 
                        HorizontalAlignment="Left"
                        />
                </ScrollViewer>
            </DataTemplate>
        </FlipView.ItemTemplate>
    </FlipView>

The code here shows that the background and image all have data bindings that correspond to the data in the XML source. Now, you need to populate these bindings with the info you have. You place the code in the cpp file of the same name. Here is the code in MainPage.cpp that helps with the data binding.

void MainPage::DocumentLoaded(_In_ Document^ document)
{
    // Parse the document into an element tree.
    document->Parse();

    auto contentRoot = document->GetContentRoot();

    if (contentRoot != nullptr)
    {
        // Create a collection of content element to bind to the view
        auto items = ref new Platform::Collections::Vector<Platform::Object^>();

        auto pageContent = contentRoot->GetFirstChild();

        while (pageContent != nullptr)
        {
            items->Append(ref new PageModel(pageContent, document));

            pageContent = pageContent->GetNextSibling();
        }

        FlipView->ItemsSource = items;

        // Load the saved document state if any
        LoadState(ApplicationData::Current->LocalSettings->Values);

        m_document = document;
    }
}

The code here parses the XML document and then creates a vector of content elements in order to bind them to the XAML elements in the UI.

Custom font loading

To use custom fonts in this DirectX app, you must implement an asynchronous font loader and collection in DirectWrite. In a DirectX app, if you want to provide a custom font to use on text elements, you need to use DirectWrite to get the new fonts that aren’t installed on the user's system. In the sample, the code that loads and makes these new fonts available is found in the FontFileStream and FontLoader classes.

The FontFileStream class is an implementation of the IDWriteFontFileStream interface and takes the actual font file and loads it into a form that the app can digest. This involves methods that handle reading file fragments, releasing file fragments, getting the file size, and getting the last edited time.

This is the code for reading and releasing the font file fragment.

HRESULT STDMETHODCALLTYPE FontFileStream::ReadFileFragment(
    _Outptr_result_bytebuffer_(fragmentSize) void const** fragmentStart,
    UINT64 fileOffset,
    UINT64 fragmentSize,
    _Out_ void** fragmentContext
    )
{
    // The loader is responsible for doing a bounds check.
    if (    fileOffset <= m_data->Length
        &&  fragmentSize + fileOffset <= m_data->Length
        )
    {
        *fragmentStart = m_data->Data + static_cast<ULONG>(fileOffset);
        *fragmentContext = nullptr;
        return S_OK;
    }
    else
    {
        *fragmentStart = nullptr;
        *fragmentContext = nullptr;
        return E_FAIL;
    }
}

void STDMETHODCALLTYPE FontFileStream::ReleaseFileFragment(
    _In_ void* fragmentContext
    )
{
}

This is the code for retrieving the size of the file and the last edited time.

HRESULT STDMETHODCALLTYPE FontFileStream::GetFileSize(
    _Out_ UINT64* fileSize
    )
{
    *fileSize = m_data->Length;
    return S_OK;
}

HRESULT STDMETHODCALLTYPE FontFileStream::GetLastWriteTime(
    _Out_ UINT64* lastWriteTime
    )
{
    // The concept of last write time does not apply to this loader.
    *lastWriteTime = 0;
    return E_NOTIMPL;
}

After all the pieces are in place and you can access the info in the font file, you must write the code that allows you to construct a custom font collection and font objects from the file stream. This code can be found in the FontLoader.cpp file.

First, you need a function that loads and enumerates the fonts that you want to use. The code here is for the LoadAsync function that looks in the fonts directory in the Magazine sample and enumerates a list of the .ttf font files in the directory.

task<void> FontLoader::LoadAsync()
{
    // Locate the "fonts" subfolder within the document folder
    return task<void>([this]()
    {
        task<StorageFolder^>(m_location->GetFolderAsync("fonts")).then([=](StorageFolder^ folder)
        {
            // Enumerate a list of .TTF files in the storage location
            auto filters = ref new Platform::Collections::Vector<Platform::String^>();
            filters->Append(".ttf");

            auto queryOptions = ref new QueryOptions(CommonFileQuery::DefaultQuery, filters);
            auto queryResult = folder->CreateFileQueryWithOptions(queryOptions);

            return queryResult->GetFilesAsync();

        }).then([=](IVectorView<StorageFile^>^ files)
        {
            m_fontFileCount = files->Size;

            std::vector<task<IBuffer^>> tasks;

            for (uint32 i = 0; i < m_fontFileCount; ++i)
            {
                auto file = dynamic_cast<StorageFile^>(files->GetAt(i));

                tasks.push_back(task<IBuffer^>(FileIO::ReadBufferAsync(file)));
            }

            return when_all(tasks.begin(), tasks.end());

        }).then([=](std::vector<IBuffer^> buffers)
        {
            for each (IBuffer^ buffer in buffers)
            {
                auto fileData = ref new Platform::Array<byte>(buffer->Length);
                DataReader::FromBuffer(buffer)->ReadBytes(fileData);

                ComPtr<FontFileStream> fontFileStream(new FontFileStream(fileData));
                m_fontFileStreams.push_back(fontFileStream);
            }

        }).wait();
    });
}

Now, write another method that takes the font collection key that is passed in when DirectWrite creates a custom font collection. This method takes this value and returns a font file enumerator.

HRESULT STDMETHODCALLTYPE FontLoader::CreateEnumeratorFromKey(
    _In_ IDWriteFactory* factory,
    _In_reads_bytes_(fontCollectionKeySize) void const* fontCollectionKey,
    uint32 fontCollectionKeySize,
    _Outptr_ IDWriteFontFileEnumerator** fontFileEnumerator
    )
{
    *fontFileEnumerator = ComPtr<IDWriteFontFileEnumerator>(this).Detach();
    return S_OK;
}

This method takes in the same collection key and returns a stream for the font file.

HRESULT STDMETHODCALLTYPE FontLoader::CreateStreamFromKey(
    _In_reads_bytes_(fontFileReferenceKeySize) void const* fontFileReferenceKey,
    uint32 fontFileReferenceKeySize,
    _Outptr_ IDWriteFontFileStream** fontFileStream
    )
{
    if (fontFileReferenceKeySize != sizeof(size_t))
    {
        return E_INVALIDARG;
    }

    size_t fontFileStreamIndex = *(static_cast<size_t const*>(fontFileReferenceKey));

    *fontFileStream = ComPtr<IDWriteFontFileStream>(m_fontFileStreams.at(fontFileStreamIndex).Get()).Detach();

    return S_OK;
}

The next 2 methods are helper methods. The first function moves to the next font file in the file stream, and the other works as a simple getter for the current font file. Here is the code for these 2 methods.

HRESULT STDMETHODCALLTYPE FontLoader::MoveNext(OUT BOOL* hasCurrentFile)
{
    *hasCurrentFile = FALSE;

    if (m_fontFileStreamIndex < m_fontFileCount)
    {
        DX::ThrowIfFailed(
            m_dwriteFactory->CreateCustomFontFileReference(
                &m_fontFileStreamIndex,
                sizeof(size_t),
                this,
                &m_currentFontFile
                )
            );

        *hasCurrentFile = TRUE;
        ++m_fontFileStreamIndex;
    }

    return S_OK;
}

HRESULT STDMETHODCALLTYPE FontLoader::GetCurrentFontFile(OUT IDWriteFontFile** currentFontFile)
{
    *currentFontFile = ComPtr<IDWriteFontFile>(m_currentFontFile.Get()).Detach();
    return S_OK;
}

With our code complete, you now have a functioning font file loader that works asynchronously with the app. These 2 classes work together to enumerate the files on the system, load them into a font file stream, and create a custom font collection with these fonts that you can use.

Best practices for using VSIS

Windows 8 supports Surface Image Source and Virtual Surface Image Source when you want to perform DirectX and XAML interoperation, but which one to use depends on what you want to do.

There are 3 main scenarios that have each have an option for XAML interoperation.

To draw an element like a texture as a bitmap in your app.
To draw a bitmap that is larger than the screen, so it requires panning.
To improve performance of touch manipulation.

Surface Image Source is good if you need to manage a static image or update the content of the SIS occasionally. A good example of when SIS is useful is if you want to draw a bitmap as a XAML UI element. This element may need to be updated to display different info, but is largely static. SIS works well because the updates that your app makes to the SIS is synched up with the XAML UI thread, so typically it works best in scenarios where it needs to be as performant as the UI in the app.

But if you need a large bitmap or DirectX content area that needs to be directly manipulated scrolled or panned) then use a VSIS. A VSIS is useful when the info you want to display doesn’t fit on the screen or element. An example of good use for a VSIS is a reading app with scrollable text, or a mapping app that requires panning and zooming to explore the map.

If none of these scenarios match your apps use case, then VSIS or SIS might not be right for your app. Specifically, if performance is a huge concern for your app, the SwapChainBackgroundPanel element in XAML may be best. For more info see Windows.UI::Xaml::Controls::SwapChainBackgroundPanel.