GPU based image processing

  • Article

Introduction

This version of the Lumia Imaging SDK supports GPU-based image processing, using Direct2D under the hood. This enables much higher performance for many tasks compared to the previous SDK version on the same device.

Resource management

When using GPU based processing, the renderer creates and owns underlying Direct2D objects and resources.

The creation of Direct2D resources is deferred until (and if) they are needed. When a renderer has been used once for GPU rendering, it must be disposed in order to free these resources. The garbage collector can also trigger disposal, but as usual it's not possible to predict when this occurs.

Two major points of consideration arise:

  • Since creating a new renderer has a cost in (CPU) performance and memory consumption, the renderer should be kept alive and reused by the app if it will be needed again soon.

  • Since keeping the renderer alive implies keeping the Direct2D objects alive, the renderer should be disposed of when it is no longer needed.

For the best results, apps must strike a balance between reusing renderers and eagerly disposing them. Where to place the emphasis depends on the overall scenario. A few examples follow:

Scenario 1: Interactive/realtime rendering.

In this scenario, a SwapChainPanelRenderer is used to update an image displayed in the user interface. The renderer is tied to a SwapChainPanel defined in the user interface.

The image is updated rapidly, up to 60 times per seconds, perhaps in response to user input.

The renderer should have the same lifetime as the SwapChainPanel (in other words, as long as the user stays in the view). The app simply calls RenderAsync on the renderer whenever an update is needed.

When the view closes, the app disposes of the renderer.

Scenario 2: "One-off" rendering.

In this scenario, the app wants to render a single image to a JPEG buffer and save it to disk.

The app creates and uses a JpegRenderer, and when the rendering is done, it disposes of it.

Note: To conserve memory, CPU rendering should be used for larger images if possible.

Scenario 3: Batch rendering.

In this scenario, the app wants to render a number of thumbnail-sized images either for present or future use.

The app creates a single BitmapRenderer. It queues up the rendering of each thumbnail in sequence, calling RenderAsync on the renderer for each thumbnail. When all the thumbnails have been rendered, the renderer is disposed.

Additional suggestion: If CPU rendering is used, more than one renderer could be used to make better use of idle CPU cores. For this to actually be an improvement depends on a number of factors, including the number of available and idle CPU cores and the amount of free memory.

GPU vs CPU scheduling

The renderer will, by default, attempt to run as much of the image processing graph as possible on the GPU, and revert back to CPU rendering when necessary. This corresoponds to having the RenderOptions property set to RenderOptions.Mixed.

To force the behavior in either direction, use the RenderOptions property of the renderer.

  • To force CPU rendering, set this property to RenderOptions.Cpu.
  • To force GPU rendering, set this property to RenderOptions.Gpu.

Note: Setting the RenderOptions property to either CPU or GPU instructs the renderer to only use the specified rendering. If any effect in the image processing graph does not support that rendering option an InvalidOperationException will be thrown during a rendering operation.

Image Sources

Source images are internally represented as Direct2D image objects when rendering on GPU. When a SoftwareBitmap, Bitmap, or IBuffer is passed to the SDK and used in GPU rendering, a corresponding Direct2D bitmap is created.

In interactive scenarios especially, the app should be aware that if a new SoftwareBitmap, Bitmap, or IBuffer instance is passed each time the result image is rendered, new Direct2D resources will be created and the image data copied.

To avoid these costs, the app is encouraged to reuse the source SoftwareBitmap, Bitmap, and IBuffer objects. When the app passes the same instance again for subsequent renders, the SDK will reuse previously created Direct2D bitmaps.

To indicate that the image contents have changed (perhaps because the app has edited the pixels of a Bitmap), the app should call the Invalidate method on the image source.

Limitations

  • The GPU rendering image processing graph uses full bitmaps of intermediate images. Therefore, memory consumption is higher compared to when tile based CPU rendering is used.
  • Source images in JPEG or other compressed format via BufferImageSource, RandomAccessStreamImageSource, StorageFileImageSource, BufferProviderImageSource, and StreamImageSource currently consume as much memory as a corresponding BitmapImageSource would.