Best practices

This section contains suggestions for getting the best results from the Lumia Imaging SDK. It covers topics like runtime performance, memory consumption, and getting the most of out of specific image sources and effects.

Contents

1. Runtime performance

In addition to the following the advice in this section, when you investigate runtime performance issues, also avoid using WriteableBitmap.

1.1. Parallelize

It's natural to write code sequentially, and the async/await pattern even promotes it. But doing so indiscriminately can lead to missed opportunities to parallelize. Doing something asynchronously doesn't automatically mean parallelism.

When you have to run multiple tasks that are independent of each other, don't just call them in sequence like this.

var result1 = await DoSomethingAsync(); 
var result2 = await DoSomethingElseAsync(); // this task won't start until the first completes!

Instead, you can start the tasks first and apply the await keyword later, like this.

var task1 = DoSomethingAsync();
var task2 = DoSomethingElseAsync(); // now this task starts right after the first call returns
// ...
await task1;
await task2;
// alternatively: await Task.WhenAll(new Task[] { task1, task2 });

1.2. Avoid temporary bitmaps

Don't render into temporary bitmaps unnecessarily if you have a pipeline with separate stages of work. Instead, passing an IImageProvider directly as a source into the next effect is faster and probably consumes less memory.

1.3. Reduce calls to GetInfoAsync

Calling GetInfoAsync determines image size and other info intelligently, but depending on the chain of effects and filters, it may mean doing significant work.

If your app has the original image size value, and if you know that the effects and filters do not alter the size, pass that value along in a variable and avoid the call to GetInfoAsync altogether.

2. Threading issues

2.1. Avoid WriteableBitmap and WriteableBitmapRenderer

For several reasons, avoid using WriteableBitmap as much as possible. Use Bitmap and BitmapRenderer instead. The exception is when you really do need a WriteableBitmap, and the reason should be that you want to display it.

  • WriteableBitmap is tied to a certain CoreDispatcher, which means only "the UI thread" is allowed to manipulate it. This includes WriteableBitmapRenderer, which must switch to the UI thread internally. This impacts app performance and responsiveness.
  • There are memory/garbage collector issues with WriteableBitmap, whereas the memory allocated by Bitmap is native memory handled by the Imaging SDK.
  • There may be extra, per-pixel conversion involved. WriteableBitmap in Silverlight on Windows Phone 8
    expects pixels in pre-multiplied alpha format (Pbgra8888).
  • Don't use a WriteableBitmap as a source image. There are memory/garbage
    collector pitfalls that are due to WriteableBitmap being tied to XAML and the .NET framework (at least in
    Windows Phone 8).

2.2. Don't call Task.Wait()

This is nearly always a bad idea. You are blocking the calling thread, which is forbidden if that thread happens to be the UI thread, and makes the Thread Pool perform worse if it's done on a Thread Pool thread.

The effects of calling Wait may not be immediately noticeable, but it's not uncommon to get deadlocks later as the app evolves.

The only situation where calling Task.Wait is suitable is when the app controls the calling thread exclusively (in other words, from a dedicated worker thread). This is not encouraged in the Windows Runtime app model.

2.3. Use 'await' optimally

One observed source of performance and stability issues has been in how apps use the await keyword. The default behavior of await can be treacherous. It's especially important to remember that after an awaited task has completed, execution resumes on the same thread that performed the await.

Deadlock risk

If the await is performed from the UI thread, deadlock can happen if the asynchronous work also involves the UI thread. Note that this circumstance may be obscured from the point of view of the app, so if deadlock happens, it can be difficult to debug.

It may be even less obvious when using the Imaging SDK, because you have a (potentially large) graph of effects and sources that are all being activated in the final call to RenderAsync on the renderer.

For instance, passing a certain stream into a StreamImageSource could mean taking a hidden dependency on the UI thread, depending on where the stream comes from. It has been observed that some framework APIs and OS facilities (like a file picker, or camera capture) that dispense streams may imply a hidden UI thread dependency. This may even vary between OS versions.

Performance loss

If several tasks are awaited in sequence, execution flow will "ping-pong" back and forth between thread-pool threads and the original calling thread (in an app, that's often the UI thread). In most cases, this is unnecessary and just slows the app down.

Thinking about execution flow

This example illustrates the execution flow of the default await behavior.

// (on the UI thread)
var result = await DoSomethingAsync(); // (work done on thread pool thread)
// (back on the UI thread)
var result2 = await DoSomethingElseAsync(result); // (work done on thread pool thread)
// (back on the UI thread)

Clearly there is more movement between threads here than is necessary.

This can be avoided by adding the option ConfigureAwait(false) to the task, as shown here.

// (on the UI thread)
var result = await DoSomethingAsync().ConfigureAwait(false); // (work done on thread pool thread)
// (remain on thread pool thread)
var result2 = await DoSomethingElseAsync(result).ConfigureAwait(false); // (remain on thread pool thread)
// (remain on thread pool thread)

Note how the execution flow changes. After each task finishes, the task scheduler picks the optimal thread to resume on--often remaining on the previous thread-pool thread. Deadlock becomes less likely, and performance is improved.

You probably don't want to remain on the thread-pool thread forever, so how do you return execution to the original calling thread? The natural approach is to encapsulate the work, like this.

public async Task<Result2> DoTwoThingsAsync()
{
    // (on the UI thread)
    var result = await DoSomethingAsync().ConfigureAwait(false); // (work done on thread pool thread)
    // (still on thread pool thread)
    var result2 = await DoSomethingElseAsync(result).ConfigureAwait(false); // (still on thread pool thread)
    // (still on thread pool thread)
    return result2;
}

// ...

// (on the UI thread)
var result2 = await DoTwoThingsAsync(); // (work done on thread pool thread)
// (back on the UI thread)

Rule of thumb

All this may seem difficult to get right, and sometimes it can be. But, here is a simple rule of thumb that rarely fails:

Almost always use ConfigureAwait(false), except when you must return to the UI thread.

The only real risk in following this advice comes when you inadvertently call a UI object from a thread-pool thread. This simply causes an exception, which is arguably easier to figure out and fix than mysterious deadlocks or performance issues caused by the default behavior of await.

3. I/O performance

3.1. Picking the right file format

When loading images from storage or a network, favor JPEG, especially if the images are large. Compressed JPEG data can be decoded and buffered partially on demand, which means memory requirements are much lower.

3.2. Start loading early

If you have "formatted" image data (like JPEG, BMP, PNG, or GIF) and you have the opportunity to load it early, do so.

  • The simplest way is to call PreloadAsync early on the image source. This allows the I/O to be overlapped with app's other work.
  • If you know you will be rendering multiple times from the same image data, consider loading the file in your app instead, and using a BufferImageSource

Note: Awaiting (using await) the result of PreloadAsync is recommended only for diagnostic purposes. A render operation that follows will internally wait for the preload to finish anyway. Redundantly awaiting it in the app is unnecessary.

3.3. Don't add overhead

If image data is already in memory (in an array or IBuffer), don't wrap it in a MemoryStream or RandomAccessStream, only to to pass it into a StreamImageSource or RandomAccessStreamImageSource. This is unnecessary, because the SDK will read the stream into an IBuffer anyway.

An array in C# can easily be turned into an IBuffer, so favor BufferImageSource over the stream and file sources whenever you have the option.

4. Memory consumption

In addition to the following the advice in this section, when you investigate memory-consumption issues, also avoid using temporary bitmaps.

4.1. Use virtual image sources

Use "virtual" image sources such as ColorImageSource and GradientImageSource instead of generated or precomputed bitmaps. Virtual image sources consume minimal memory by generating their pixels on the fly, rather than serving them from a bitmap image in memory.

4.2. Dispose of resources

For SDK objects that implement the IDisposable interface, call the Dispose method
as soon as the object is no longer needed. When applicable, using statements can be nested in a structured way, like this.

using (var source = new StorageFileImageSource(storageFile))
using (var effect = new FilterEffect(source))
using (var renderer = new JpegRenderer(effect))
{
  // apply effects here
  var buffer = await renderer.RenderAsync().AsTask().ConfigureAwait(false);
  SaveBuffer(buffer);
}

Of course, creating and destroying the image effect graph also comes with a runtime performance cost, so consider carefully how your app will use the SDK objects.

It's common for an app to need both approaches, where the "preview pipeline" keeps the SDK objects alive just to be able to call RenderAsync to update the displayed preview image. Because the preview uses only display-sized images, this keeps the memory footprint manageable. The app may also have a "save pipeline" that works with the full-sized image, and here it's more important to use Dispose carefully.

5. Effect-specific suggestions

5.1. LensBlurEffect: Set the Quality

Lens blur can often be used at a lower Quality setting without noticeable effect. Lowering the quality value reduces the data set that the effect has to work on, which improves performance and reduces memory consumption. Especially for larger sources, running the effect with a quality value of 1.0 might not be possible, and an OutOfMemoryException will be thrown.

The amount of memory the lens blur effect consumes depends on the input source size, so adjusting the quality might be necessary even when other effects are successfully applied to the same source.

Consider using a lower quality level when rendering a screen-sized preview, and a higher quality level when rendering the final result to storage.

5.2. InteractiveForegroundSetmenter: Set the Quality

Interactive foreground segmentation is an expensive operation that can usually run with a lower Quality setting without a noticeable effect on the resulting mask. Lowering the quality value reduces the data set that the effect has to work on, which improves performance and reduces memory consumption. Especially for larger sources, running the effect with a quality value of 1.0 might not be possible, and an OutOfMemoryException will be thrown.

The amount of memory the interactive foreground segmenter consumes depends on the input source size, so adjusting the quality might be necessary even when other effects are successfully applied to the same source.

Consider using a lower quality level when rendering a screen-sized preview, and a higher quality level when rendering the final result to storage.

5.3. Various: Call Invalidate

Note: Available since version 1.2 beta.

Some image sources, such as the BitmapImageSource and the BufferImageSource, have an Invalidate method that can be used to inform the source that it needs to reload its content. If you alter the content in the IBuffer or Bitmap of such a source without changing any other property, call Invalidate before the next render or load operation. Otherwise the new content might not be used.

Show: