Custom image sources, effects, and effect groups
The Lumia Imaging SDK can be extended by the app developer. Custom image sources, effects, and effect groups can be created by implementing public interfaces.
- Image Sources
- Effects
- CPU Image Workers
- Direct2D Image Workers
- Effect Groups
This document details how to go about implementing these extended types.
Image Sources
All image sources must implement the following interfaces:
- IImageProvider
- IImageProvider2
- IImageResource or IAsyncImageResource (depends on the nature of the image)
C# (.NET)
Users of C# or another .NET language can use the ImageSourceBase class found in Lumia.Imaging.Managed.dll. This base class handles most of the plumbing, leaving mostly the image processing itself to the app developer.
The following example shows a basic C# image source that inherits ImageSourceBase.
class SimpleImageSource : ImageSourceBase
{
private uint[] m_colors;
public SimpleImageSource(Windows.Foundation.Size size, Windows.UI.Color[] colors) :
base(size)
{
// Convert colors to an array of uint for faster processing inside OnProcess().
m_colors = colors.Select( color => FromColor(color) ).ToArray();
}
protected override void OnProcess(PixelRegion pixelRegion)
{
pixelRegion.ForEachRow((index, width, pos) =>
{
for (int x = 0; x < width; ++x, ++index)
{
// Pick one of the configured colors based on the x-coordinate.
var color = m_colors[x % m_colors.Length];
pixelRegion.ImagePixels[index] = color;
}
});
}
protected override IImageProvider2 Clone()
{
// Clone this object, making sure to initialize the clone with the relevant state.
return new SimpleImageSource(ImageSize);
}
}
Performance notes
The example source code is written to be explanatory, not for optimal performance.
- Prefer a native implementation (C++ with WRL or C++/CX) for the best overall performance.
- Using PixelRegion.ForEachRow simplifies iterating over the pixels, but using a for-loop can be faster.
- Using the FromColor method simplifies converting a Windows.UI.Color to the uint expected in the PixelRegion.ImagePixels array, but using bit operations to merge color components will be faster.
C++/CX (and WRL)
Users of C++/CX and WRL should implement the interfaces directly.
One way to create such an image source is to keep a BitmapImageSource as a member variable, and forward most calls to that object. The user class acts like a wrapper, and should implement the same interfaces that BitmapImageSource does.
Effects
For detailed sample implementations of Effects, see CustomEffectSample in the GitHub samples repository.
All effects must implement the following interfaces:
- IImageProvider
- IImageProvider2
- IImageConsumer
- IImageConsumer2
Effects are essentially factories for image worker objects (implementing IImageWorker). The renderer may schedule image processing work to run on the CPU or GPU depending on the supported render options in the effect and the structure of the processing graph.
Since an effect is free to support either CPU rendering, Direct2D (GPU) rendering, or both, it reports the supported types of image worker in the property IImageProvider2::SupportedRenderOptions. The IImageProvider2::CreateImageWorker method is then called during rendering to create image workers of the supported types.
For the most obvious use case it is enough to implement only one type of the worker object, and indeed we expect that most developers will choose to implement their custom effect only once, either on CPU or GPU. That said, there are scenarios where it might be necessary to implement both. For instance, you could be limiting rendering to CPU only on low-end devices to satisfy memory constraints, but want to take advantage of hardware acceleration on high-end devices.
Another reason is that you might be applying your effect to very large images, larger than the largest 2D texture size the GPU can handle. If the renderer is set to RenderOptions.Mixed then the SDK will fall back on CPU processing whenever the image is too large for the GPU to handle.
CPU Image Workers
Image workers that perform work on the CPU must implement ICpuImageWorker.
Direct2D Image Workers
Image workers that perform work using Direct2D (on the GPU) should implement IDirect2DImageWorker or IDirect2DPixelShaderImageWorker.
C# (.NET)
Users of C# or another .NET language can use the EffectBase class found in Lumia.Imaging.Managed.dll. This base class handles most of the plumbing, leaving mostly the image processing itself to the app developer.
internal class HalfBrightEffect : EffectBase
{
public HalfBrightEffect(IImageProvider source)
{
Source = source;
}
public override IImageProvider2 Clone()
{
return new HalfBrightEffect(((IImageProvider2)Source).Clone());
}
public override RenderOptions SupportedRenderOptions
{
get
{
return RenderOptions.Cpu; // This example supports only CPU based rendering.
}
}
public override IImageWorker CreateImageWorker(IImageWorkerRequest imageWorkerRequest)
{
if (imageWorkerRequest.RenderOptions == RenderOptions.Cpu)
{
return new BlockBasedWorker();
}
return null; // Unsupported requests get null as the return value.
}
// This is the image worker implementation, which performs actual processing.
private class BlockBasedWorker : CpuImageWorkerBase
{
public BlockBasedWorker()
: base(new[] {ColorMode.Bgra8888}) // Supported color modes.
{
}
protected override void OnProcess(PixelRegion sourcePixelRegion, PixelRegion targetPixelRegion)
{
targetPixelRegion.ForEachRow((index, width, position) =>
{
for (int i = 0; i < width; ++i)
{
var c = sourcePixelRegion.ImagePixels[index + i];
c >>= 1;
c &= 0x7F7F7F7F;
targetPixelRegion.ImagePixels[index + i] = c;
}
});
}
}
}
Performance notes
The example source code is written to be explanatory, not for optimal performance.
- Prefer a native implementation (C++ with WRL or C++/CX) for the best overall performance.
- Using PixelRegion.ForEachRow simplifies iterating over the pixels, but using a for-loop can be faster.
C++/CX (and WRL)
Users of C++/CX and WRL should implement the interfaces directly.
Effect Groups
Encapsulating multiple effects in one "group" effect is possible. Such a class basically looks like an Effect, but behaves differently when IImageProvider2::Clone is called.
C# (.NET)
Users of C# or another .NET language can use the EffectGroupBase base class found in Lumia.Imaging.Managed.dll. This base class handles most of the plumbing, so that the user defined class can focus on the structure of the effect group itself. The resulting class can be used directly as if it's a single effect.
class BrightGrayEffect : EffectGroupBase
{
private BrightnessEffect m_brightnessEffect;
private GrayscaleEffect m_grayscaleEffect;
public BrightGrayEffect()
{
m_brightnessEffect = new BrightnessEffect();
m_grayscaleEffect = new GrayscaleEffect(m_brightnessEffect);
}
public double Brightness
{
get { return m_brightnessEffect.Level; }
set { m_brightnessEffect.Level = value; }
}
protected override IImageProvider PrepareGroup(IImageProvider groupSource)
{
// Connect the group source to the "input" effects.
m_brightnessEffect.Source = groupSource;
// Return the "output" effect.
return m_grayscaleEffect;
}
}
C++/CX (and WRL)
Users of C++/CX and WRL should implement the interfaces directly as if were implementing an Effect. However, no image worker needs to be returned.
In the implementation of IImageProvider2::Clone, instead of cloning itself as a regular Effect would, the Effect Group clones the child effects in the sub-graph and connects them into the resulting processing graph.
The end result is that after cloning, the Effect Group is replaced with the sub-graph that it represents.